(MENAFN- Ameliorate Digital Consultancy) The Smart Waste Bins Market report offers qualitative and quantitative insights as well as a thorough examination of the market size and expansion rate of ...

Smart Waste Bins Market Will See Strong Expansion Through 2029|Bine,Enevo,Evreka Themarketpublicist

Spot the ten places in which the right image differs from the left one. Only the right image has the clickable areas. The left image is a screen shot of Sporcle's new construction image seen during site maintenance. Picture Click: Click on regions of an ...

Diffs: Spot Ten Sporcle Construction Issues/Differences

♻️ It's time for the weekly news! Welcome back to Evreka's Sustainability Bulletin :) We will be here every Thursday with developments from around the world. Stay tuned and have a Day for a ...

Sustainability Newsletter

The smart waste management market size is expected to grow by USD 2.83 billion between 2022 and 2027. However, the growth momentum of the market will progress at a CAGR of 8.96% during the forecast period.

Smart Waste Management Market to grow by USD 2.83 billion during 2022-2027 | Increasing urbanization to drive the growth - Technavio

Whether clues are worded on a personal level (ie as if only one person fitted) or not, it is NEVER an indication of how many it applies to, but a way to shorten the clue.

Harry Potter: Good at teaching DADA Venn

Sardunya Otel is located in Bozcaada. Gestas Bozcaada Ferry Terminal and Amadeus Winery are worth checking out if an activity is on the agenda, while those wishing to experience the area's natural beauty can explore Çayır Beach and Ayazma Beach. Talay ...

Sardunya Otel

If you end up not knowing which smaller to miniature country that you haven't yet named: Look not for the green remaining country, but for the country name(s) NOT shown in the vicinity of where the country is. For example, similar to if you are missing San ...

Reveal the Countries of Asia

If you end up not knowing which smaller to miniature country that you haven't yet named: Look not for the green remaining country, but for the country name(s) NOT shown in the vicinity of where the country is. For example, similar to if you are missing ...

Reveal the Countries of Africa

Reveal the Countries of Europe

For the purpose of this quiz the Nordic countries are the five (main) countries that are considered as part of this region in this region. (Check the tags of this quiz for a list of them, if you are uncertain of which they are.) Further the quiz tests ...

North to South: Nordic countries in Order

Evreka

evreka

Movies, cartoons, and anime have long warned us about numerous realistic apocalyptic scenarios due to climate change that are just around the corner. From “Wall-E” to “Interstellar” and “Nausicaä of the Valley of the Wind”. But I’ve never paid much thought to it, that is until I’ve watched a Netflix documentary . “David Attenborough: A Life on Our Planet” Shocking details of our world melting and the planet’s ecosystem collapsing because of human actions made me acutely aware of the situation we’re in and that the next ten years are crucial for Earth. We produce more than we can consume and we’re drowning in trash. CO2 emissions are rising, the Earth is getting warmer, and the diversity of species is disappearing. It all made me wonder what can be done about it and how emerging technologies can help. So here are 5 IoT solutions that are helping save our planet right now. 1. Walmart’s ‘Eden’ Suite to Eliminate Food Waste Let’s start with food. Although the planet’s population is increasing drastically and the need for food is bigger than ever before, the topic of food waste has not vanished. Just imagine – a third of all food produced globally is annually. It is around 1.3 billion tons of food. Unless the problem of food waste is tackled, this number will grow . And to think that to produce a single food item many precious resources are spent, it is abominable to allow one-third of it to be discarded. wasted by 70 percent by 2050 Digital technology comes to the rescue. With digital capabilities, we can identify, prevent and/or reduce possible waste. One such example is an IoT solution devised by U.S. store chain Walmart that tracks the freshness of food as it moves from farms to the stores. ‘Eden’ The key indicator of food decay is changed levels of moisture in products due to higher temperature levels in container trucks. Using connected sensors, machine learning, and product tags for moisture sensing, the solution can trace food quality in real-time. So what happens if apples start to go bad on their way to some distant store? They are re-routed immediately to a closer store. Eliminating waste with this technology, Walmart aims to save over the next five years. two billion dollars 2. An eco-friendly way to grow food with vertical farming Growing enough food to feed all the planet’s population is another problem facing humanity. Despite a slow-down in population growth, the UN projects that the number of people living on Earth in 2050 will reach . According to the , 56% more food should be produced to feed everyone. 9.7 billion World Resources Institute Here’s where sustainable agriculture and farming could help. And no one does it better than innovators from Wageningen University in the Netherlands. In terms of gross value, this country constantly ranks among the , although you can drive across the whole of the Netherlands from the north to the south in less than four hours. world’s top agricultural exporters Their sustainable and well-controlled approach to vertical farming – that is growing crops on layers one above the other – does not rely on soil, sunshine, water, and pesticides. And more importantly, vertical farms do not require vast amounts of arable land, compared to conventional farming, and avoid further deforestation. IoT makes vertical farms more modular and helps monitor the health of crops that are grown. Smart greenhouses have sensors installed that gather data on temperature, humidity, pH, and CO2 levels, as well as light and water. Thanks to this real-time data farmers can intelligently monitor and control their crops, regulating temperature, humidity, lighting, spraying, irrigation, and more. 3. Evreka solution for a smart way to collect waste Now is the time to have the trash talk. Not only it is contaminating oceans and soils, producing harmful gasses, and making our planet inhabitable, but the whole process of garbage collection is far from perfect. To keep our cities clean and save the time and effort of waste management companies, IoT route optimization systems have been devised. One of them is a Turkey-based company called . Evreka This IoT management system makes dumpsters “smart” to gain information about how much they are filled. Sensors that are durable against heat and water are attached to containers. They collect information about location, temperature, and fill levels, and send it to the cloud. Later, this data is used to form the optimum route of the vehicle collecting garbage and to calculate the frequency of such visits. 4. Smart waste management system Envac to dispose of waste One Swedish waste management company has gone further than just making their garbage bins smart. They have implemented a vacuum-powered system that is my absolute favorite solution not only to dispose of waste but also to sort it for further reuse and recycling. This brilliant waste management system is called . It uses high-pressure underground tubes that suck garbage bags from the bins and transport them into a sorting center. But before making it to the tubes, the garbage bag has to be thrown in the correct container – for organic waste, recyclables, and for all other waste. Organic waste is turned into biogas and all other types of waste are used to produce electricity or heat. Envac The bins are equipped with sensors that detect when they are full. It takes 10 seconds to empty the whole bin underground so that they don’t overflow. One set of such smart waste bins can serve 30-50 families. Residents even get feedback on their smartphones about how much waste they produce and how much biogas. 5. Energy-efficient cities with Novalume's smart streetlight solution City street lights are the last thing I’d like to mention here. Despite using lots of energy and producing air pollution with CO2 emissions, city lights can have negative effects on human health. There are companies with ready-made IoT solutions that help not only save costs with LED streetlights but also reduce energy consumption and CO2 emissions by 30%. is a Danish company that delivers lighting solutions for city municipalities. Their IoT system called LUMINTELL™ optimizes street light usage to decrease energy use during off-peak traffic hours. Each street light is wirelessly connected to the online application in the cloud. Using the software solution, municipalities workers can see the entire street light installation and monitor their operations. This function reduces 75% of maintenance costs. Novalume Solutions smart city Energy-efficient solutions, solutions that help recycle waste with sensor-based technology, vertical farming, and solutions that help eliminate food waste – all these examples of IoT technology solutions are great tools in our battle to save Earth from ecological catastrophes that are looming. But the more such solutions, the better. So, now it’s up to us, humans, to use all methods available to us and act.

A Battle for Earth: 5 Ways IoT Can Help Save Our Planet

Hi! I’m Nikolai Khechumov from Avito’s Application Security team. This is the second part of our journey, where we are trying to improve secrets detection. In the previous part, we examined different types of secrets, understood the core problems, and hit a dead end. Now we are going to make a breakthrough and do something useful. Imitating SAST Okay, since we cannot easily use the existing tools, let’s try to understand how they work and review the first two steps any SAST tool makes to build an AST. A very simplified scheme is shown below: These two steps are “lexing” and “parsing.” The Lexing (also known as ‘tokenization’) stage receives a code as just a stream of chars. Then it finds very basic language-specific syntax constructions and outputs them as a set of typed tokens — small strings with semantic interpretation. For example, a token ‘def’ is not just three chars but a Python keyword reserved for function declaration. We now have valuable insights about the purpose of every token, but the context itself is still missing. The parsing stage adds valuable context. It combines tokens into higher-level language constructions and outputs them in a form called an “abstract syntax tree.” Now we have more structural details: groups of tokens became variables, functions, and so on. For us, the question is still the same: how. Eureka! One day, while making a presentation for a talk, I had a code snippet. I wanted to format it nicely, started a browser, and typed “syntax highlighting online.” Then I opened the first link, pasted my code, pressed “highlight,” and.. a lighting strike hit somewhere near me. What I did is I found a tool that: definitely makes tokenization under the hood
light enough to work inside a web browser
supports dozens of languages and formats out of the box Forget about the presentation. Let’s find something similar running on Python. And yes, I found it. 538 languages supported I found Pygments, a fantastic library that solved the most significant problem: language-dependent tokenization. It still uses regexes under the hood, but those regexes are not written by me. That’s a killer feature! The library is about syntax highlighting but it features RawTokenLexer, so we can output a raw stream of tokens with their semantic meaning. Our first problem — decreasing the number of strings for analysis — is solved. Now we understand the type of every token and can just ignore useless ones: keywords, punctuation, numbers, operators, etc., leaving only literals and comments. But we are still unable to understand the names and values of variables. Variable Detection The lexing problem seems to be solved. Now let's move on to parsing. Building a true AST looks a bit redundant for our problem, so let’s optimize. Further research has shown that the type of token is more important than its value for variable detection. We need to look for patterns inside a stream of token types to detect variables. Once the pattern is found, we can make additional checks against the values of the tokens inside it. To create a Token Type Stream (TTS), I took one character from the type name of each token. Thanks to Pygments, token types are common across languages and formats. Eventually, a Variable Detection Rule (VDR) may contain the following: Reference pattern to look inside a TTS (regex with match groups)
MatchRules for match groups that check for expected values inside the pattern (eq. useful for finding assignment punctuation)
MatchSemantics to clarify a group’s semantic purpose Example! So let’s take our simple code snippet: def main(arg: str):
  a = 3 
  b = 'hello'
  return f'{a}{b}' Represent as an array with TTS. Basically, the variable of our interest (b) is between the 20th and the 25th token. The reference pattern here should be something like this: (n)t*(o|p)t*(s)(s)(s)t Let’s apply it to the TTS Match groups help us to apply additional verification logic. And finally, a rule may be supplemented with our vision of semantics. Summarizing everything The cool part here is that approach allows you to cover any language with variable detection using only 4-6 rules. New Secrets Detection Rule Semantic information about a given file enables us to analyze deeper in the right context. For example, we can now calculate entropy for a variable’s value, knowing for sure that this specific substring is a variable's value. Names of variables are also important for analysis. There are two new rules I introduced: Suspicious variable naming (low confidence)
Suspicious variable naming + high entropy of a variable's value Now we can also use hashed secrets as a source of new rules. A hashed secret rule can contain information about the original secret length to improve performance. Outcomes This approach became a giant step forward for Avito: we received a noticeable performance boost (up to 60%) and +200% new findings we could not see before. Of course, the false positive rate has also grown, but it’s mostly due to test secrets. The findings are semantically correct. Limitations Plaintext files without extensions or incorrect extensions are obviously unsupported; tokenization fallbacks to the “blind mode”
Variable Detection Rules (VDRs) are still a kind of compromise; they’re language-specific and may not guarantee absolute correctness (although we did not face any problem with that) We’re ready to share it with you! Our code scanning model requires every scanner to be a web service that supports our internal protocol, so we decided to extract the core functionality and open-source it as a CLI tool called DeepSecrets. You can find it here: https://github.com/ntoskernel/deepsecrets Release notes are in a separate article here. Thank you! The featured image for this piece was generated with stable diffusion v2.1
Prompt: Illustrate a computer screen with the caution symbol. Hi! I’m Nikolai Khechumov from Avito ’s Application Security team. This is the second part of our journey, where we are trying to improve secrets detection. In the previous part , we examined different types of secrets, understood the core problems, and hit a dead end. Now we are going to make a breakthrough and do something useful. Avito Application Security In the previous part Imitating SAST Okay, since we cannot easily use the existing tools, let’s try to understand how they work and review the first two steps any SAST tool makes to build an AST. A very simplified scheme is shown below: A very simplified scheme is shown below: These two steps are “lexing” and “parsing.” These two steps are “lexing” and “parsing.” The Lexing (also known as ‘tokenization’) stage receives a code as just a stream of chars. Then it finds very basic language-specific syntax constructions and outputs them as a set of typed tokens — small strings with semantic interpretation. For example, a token ‘def’ is not just three chars but a Python keyword reserved for function declaration. We now have valuable insights about the purpose of every token, but the context itself is still missing. Python The parsing stage adds valuable context. It combines tokens into higher-level language constructions and outputs them in a form called an “abstract syntax tree.” Now we have more structural details: groups of tokens became variables, functions, and so on. For us, the question is still the same: how. Eureka! One day, while making a presentation for a talk, I had a code snippet. I wanted to format it nicely, started a browser, and typed “syntax highlighting online.” Then I opened the first link, pasted my code, pressed “highlight,” and.. a lighting strike hit somewhere near me. What I did is I found a tool that: definitely makes tokenization under the hood light enough to work inside a web browser supports dozens of languages and formats out of the box definitely makes tokenization under the hood light enough to work inside a web browser supports dozens of languages and formats out of the box Forget about the presentation. Let’s find something similar running on Python. And yes, I found it. 538 languages supported I found Pygments , a fantastic library that solved the most significant problem: language-dependent tokenization. It still uses regexes under the hood, but those regexes are not written by me. Pygments That’s a killer feature! The library is about syntax highlighting but it features RawTokenLexer , so we can output a raw stream of tokens with their semantic meaning. RawTokenLexer Our first problem — decreasing the number of strings for analysis — is solved. Now we understand the type of every token and can just ignore useless ones: keywords, punctuation, numbers, operators, etc., leaving only literals and comments. But we are still unable to understand the names and values of variables. Variable Detection The lexing problem seems to be solved. Now let's move on to parsing. Building a true AST looks a bit redundant for our problem, so let’s optimize. Further research has shown that the type of token is more important than its value for variable detection. We need to look for patterns inside a stream of token types to detect variables . Once the pattern is found, we can make additional checks against the values of the tokens inside it. patterns inside a stream of token types to detect variables To create a Token Type Stream (TTS), I took one character from the type name of each token. Thanks to Pygments, token types are common across languages and formats. Eventually, a Variable Detection Rule (VDR) may contain the following: Eventually, a Variable Detection Rule (VDR) may contain the following: Reference pattern to look inside a TTS (regex with match groups) MatchRules for match groups that check for expected values inside the pattern (eq. useful for finding assignment punctuation) MatchSemantics to clarify a group’s semantic purpose Reference pattern to look inside a TTS (regex with match groups) pattern MatchRules for match groups that check for expected values inside the pattern (eq. useful for finding assignment punctuation) MatchRules MatchSemantics to clarify a group’s semantic purpose MatchSemantics Example! So let’s take our simple code snippet: def main(arg: str):
  a = 3 
  b = 'hello'
  return f'{a}{b}' def main(arg: str):
  a = 3 
  b = 'hello'
  return f'{a}{b}' Represent as an array with TTS. Basically, the variable of our interest ( b ) is between the 20th and the 25th token. b The reference pattern here should be something like this: (n)t*(o|p)t*(s)(s)(s)t (n)t*(o|p)t*(s)(s)(s)t Let’s apply it to the TTS Match groups help us to apply additional verification logic. And finally, a rule may be supplemented with our vision of semantics. Summarizing everything Summarizing everything The cool part here is that approach allows you to cover any language with variable detection using only 4-6 rules. New Secrets Detection Rule Semantic information about a given file enables us to analyze deeper in the right context. For example, we can now calculate entropy for a variable’s value, knowing for sure that this specific substring is a variable's value. Names of variables are also important for analysis. There are two new rules I introduced: Suspicious variable naming (low confidence) Suspicious variable naming + high entropy of a variable's value Suspicious variable naming (low confidence) Suspicious variable naming (low confidence) Suspicious variable naming + high entropy of a variable's value Suspicious variable naming + high entropy of a variable's value Now we can also use hashed secrets as a source of new rules. A hashed secret rule can contain information about the original secret length to improve performance. Outcomes This approach became a giant step forward for Avito: we received a noticeable performance boost (up to 60%) and +200% new findings we could not see before. Of course, the false positive rate has also grown, but it’s mostly due to test secrets. The findings are semantically correct. Limitations Plaintext files without extensions or incorrect extensions are obviously unsupported; tokenization fallbacks to the “blind mode” Variable Detection Rules (VDRs) are still a kind of compromise; they’re language-specific and may not guarantee absolute correctness (although we did not face any problem with that) Plaintext files without extensions or incorrect extensions are obviously unsupported; tokenization fallbacks to the “blind mode” Variable Detection Rules (VDRs) are still a kind of compromise; they’re language-specific and may not guarantee absolute correctness (although we did not face any problem with that) We’re ready to share it with you! Our code scanning model requires every scanner to be a web service that supports our internal protocol, so we decided to extract the core functionality and open-source it as a CLI tool called DeepSecrets. You can find it here: https://github.com/ntoskernel/deepsecrets https://github.com/ntoskernel/deepsecrets Release notes are in a separate article here. Thank you! The featured image for this piece was generated with stable diffusion v2.1 Prompt: Illustrate a computer screen with the caution symbol. The featured image for this piece was generated with stable diffusion v2.1 Prompt: Illustrate a computer screen with the caution symbol.

Modernizing Secrets Scanning: Part 2–the Semantic Eureka

The New York “BitLicence” kills innovation and makes people less safe. We know this is a fact from the story of the incandescent light bulb The venerable has now been in many countries in the west. The reasoning behind this ban was that the bulbs are ‘inefficient’, and that removing them from use will save energy and reduce the amount of ‘carbon’ that is released into the atmosphere by the people who use them. incandescent light bulb banned As a replacement for these bulbs, ‘energy efficient’ fluorescent light bulbs were planned to completely replace the incandescent light bulb, by the force of law. All of the production lines that used to manufacture the incandescent light bulb have either been stopped or are in the process of being stopped. Capital has been diverted to the production of fluorescent light bulbs, and manufacturing capacity of the factories that make them has been increased to meet the demand caused by the ban on incandescent bulbs. The incandescent light bulb is an old technology, developed and by , and refined over many decades. They are inexpensive to manufacture, made of simple, 100% recyclable non toxic parts (glass, steel and tungsten), and there are literally of receptacles that have been designed to accommodate their shape. patented Thomas Edison billions Edison’s Light Bulbs are very simple, and 100% recyclable. The new ‘Environmentally Friendly’, ‘energy saving’ are expensive to manufacture, have plastic parts, are not simple in design and contain deadly poisons like , making it necessary to dispose of them carefully, lest the mercury escape, polluting the environment . Compact Fluorescent Light Bulbs mercury and poisoning people The new bulbs also produce a hideous, unpleasant light that flickers at the frequency of the electricity mains. These bulbs have been demonstrated to have deleterious effects on people who are sensitive to their light, causing them and . migraine headaches eye strain Just before the ban on the incandescent light bulb came into force, the following announcement of a scientific breakthrough was made: Boffins: Ordinary lightbulbs can be made efficient, cheaply Incandescents nearing extinction: Impeccable timing, everyoneBy Lewis Page Posted in Physics, 1st June 2009 11:03 GMT Just as authorities in much of the Western world have moved to phase out the incandescent lightbulb, American boffins believe they have developed a process which can make the oldschool lights more efficient than energy-saving lamps. Optics boffins at the Rochester Uni in New York state say they’ve developed a process in which an ordinary lighbulb is zapped with a femtosecond-long pulse of extremely high-energy laser light. The laser blast travels through the glass to hit the tungsten filament, causing complex nano- and micro-structures to form on its surface. Once the lasered light bulb is than powered up, according to the Rochester scientists, it emits a lot more light for the same energy compared to an untreated bulb — equivalent to 40 per cent energy savings. The process of lasing incandescent bulbs wouldn’t be expensive, apparently, so they’d remain cheap compared to fluorescent energy-saving jobs. According to Rochester Uni: The process could make a light as bright as a 100-watt bulb consume less electricity than a 60-watt bulb while remaining far cheaper and radiating a more pleasant light than a fluorescent bulb. Despite the incredible intensity involved, the femtosecond laser can be powered by a simple wall outlet, meaning that when the process is refined, implementing it to augment regular light bulbs should be relatively simple. It seems that Professor Chunlei Guo of Rochester hit upon the idea of brightening-up lightbulb filaments following earlier experiments in which he and his team used laser zapping to turn metals completely black. This worked so well that Guo and his cohorts wondered if they could reverse the process. “We fired the laser beam right through the glass of the bulb and altered a small area on the filament,” says the prof. “When we lit the bulb, we could actually see this one patch was clearly brighter than the rest of the filament, but there was no change in the bulb’s energy usage.” It seems that Guo and his team of lightbulb-blasting boffins can also produce other strange effects, getting incandescent bulbs to emit partially polarised or differently-coloured light — without the energy-wasting filters that would normally be necessary. It’s the efficiency-enhancement aspect of the studies which could make headlines, however. Both the US and European Union governments are now committed to firm timetables which will see incandescent bulbs phased out in favour of more energy-efficient alternatives, such as fluorescents. This is being done in order to save energy and so lower carbon emissions. But if it’s as simple as Guo suggests to enhance an incandescent with his laser process, this may turn out to have been an unnecessary or even retrograde step. Guo’s research has been accepted for publication by the journal Applied Physics Letters, but isn’t out yet. In the meantime, there’s a pop-sci release from the university here. http://www.theregister.co.uk/2009/06/01/light_bulb_laser_blast_enhancement/ This is a perfect example of why the State should have in what technology firms must use to produce their goods and which goods people can and cannot have access to. no say In their zeal to ‘protect the environment’ the State has diverted capital and resources away from the well established incandescent light bulb production lines, , to the new fluorescent light bulb lines, that have had their manufacturing capacity ramped up in order to meet the artificially stimulated demand for the new bad bulbs. by force The decision to ban the incandescent lightbulb was made with all of the information the legislatures had to hand; i.e. all the relevant facts about the different types of bulb that were available in 2007/2008. What they did not and could not take into account was the research in the above article, which, had they known about it, may have prevented them from legislating for the complete ban. The Knowledge Problem This is the central problem with the state interfering with technology; Now that the law is in force, we are in a situation where capital has been wasted and misdirected, resources wasted and misdirected and depending on the state of the decommissioning of the incandescent bulb lines, no costless way back to the manufacturing of incandescent light bulbs. no one can predict the future. The state, by its nature, is . They cannot predict the future and they are not omniscient. In order to be able to legislate effectively, especially where technology is concerned, they would need to be omniscient, with perfect knowledge of every piece and field of ongoing research and technology, and the of each piece and field of research and technology. incompetent potential The enormity of this amount of knowledge is beyond the capacity of any man or group of men; it would mean being able to apply each existing technology (implemented or not) and each piece of research against each other (in the case above, femtosecond lasers, metallurgy and incandescent light bulbs), considering the effect of multiples of them upon each other in succession, and then considering the knock on effects of each of these, say, ten levels down on the tree. This would produce a multi dimensional matrix or tree with a size bigger than the universe. Only then would they be able to create an optimal plan that would maximize the production and movements of capital to reach any particular goal, and of course, who has the moral authority to choose the appropriate goals is something to consider. Applying a femtosecond laser to incandescent light bulbs is one solution that has produced significant increases in efficiency. Who knows what other treatments, changes in filament formulation, the glass enclosure etc. will produce? Certainly, the state cannot possibly predict them. What the state_can_ do however, is prevent research and innovation, destroying potential breakthroughs, efficiencies, savings and progress. The state is a bumbler, a reactive and ignorant dealer; they are like a cave man being tasked to turn off a diesel engine that has been left running. Instead of getting into the car by actuating the door handle and turning the ignition key counter clockwise, the caveman takes his club out and beats against the hood covering engine until it stops running. Of course, he claims success when it stops, when in fact the tank has run out of gas and the engine cuts out of its own accord. But I digress. This process, an inexpensive, and easy addition to any incandescent light bulb production line, would have saved billions in electricity bills, spared the environment from tonnes of mercury contamination, eliminated the need to build light bulb recycling plants and spared the health of many millions of people, with all the costs attendant on that. Then you must factor in are all the other applications of this femtosecond laser technology that are now going to be delayed or which will not now come into existence. femtosecond laser Think about it; the widespread deployment of femtosecond flashers would mean designing modular systems for resale to manufacturers; out of that design and manufacturing process, other processes will have emerged. Secondary uses of these flashers would have been subsequently discovered, which would have other knock on effects. The innovation cascade resulting from this process is what the state has destroyed. When you apply this example to any other industry or technology where the state legislates, and take into account the default incompetence that inheres in the people that make these decisions, you can begin to get a glimpse of the suboptimal world that we are now in. Hovercars Imagine what sort of world it would be had the state not interfered in any way with any technology. To put it into perspective, think of the ubiquity of cheap mobile phones, and then apply that example to every expensive technology that the state has controlled in any way… like the automobile. But but but… it has no wheels! “Where we’re going, we don’t need roads” or wheels! — Back to the Future Imagine how much more efficient, inexpensive, clean and beneficial cars would be had the state stayed out of the business of the details of car manufacture. ‘Consumer advocates’ would tell you that cars are safer now only because the state intervened in the manufacturing process. This may or may not be the case, but what is certain is that cars would be had the state not interfered with the manufacture of cars. safer than they are now I do not know of a single person who would not pick a safer car over an unsafe one, and since competition is fierce in car manufacture, this fact would be taken into account at every point the design stage, producing just the sort of cars that people want, . Think iPhone here; once you see it, you want it; you did not know that you would want it before it existed, you and all of the other phone manufacturers other than Apple could not even imagine it, but now that you have seen it, you want it more than … the same could be said for the very idea of the cellular phone itself. and cars that people did not know that they wanted any other mobile phone Extrapolating from all of this, it is clear that in many aspects of the way we live, we are existing in a world that is grossly distorted and sub-optimal. This world could be better in every way by orders of magnitude had the destructive and disruptive state not interfered with the innovations and interactions of men. How this applies to Bitcoin Everything that I have written here applies to Bitcoin, and the State’s race to regulate it. They will kill Bitcoin innovation just as they did in light bulb innovation, and the long term effects will be detrimental to all mankind. Innovations like will be abandoned as unnecessary, since the State will be there as guarantor. As we all know with the bank failures and regular stock market crashes, the State is useless at regulating the economy; they will damage Bitcoin if they are allowed to touch it. Bitcoin Multisig Transactions The New York BitLicense is going to cost anyone incorporated there $2,000,000 to receive a license, and then $1,000,000 per annum to stay in business thanks to imposed “compliance” rules. None of these rules will aid a single customer or cause them to be reimbursed should something go wrong. Every company submitting to BitLicense will pass compliance costs on to their customers, making Bitcoin more expensive to use for them The companies suffering BitLicense will have two choices; leave New York entirely, or lobby for every Bitcoin company everywhere to suffer the same crippling restrictions and costs to, “level the playing field”. and only them. Leaving New York will cost far more than lobbying the State to put restrictions on all Bitcoin companies, so you can bet that this will be the preferred . If the recent efforts to pare down the BitLicence succeed, the incentive to make others comply with what BitLicence proponents believe is “reasonable” by force will be strong. They have already done the hard work, why should others get off Scott free? This is the State, pitting entrepreneur against entrepreneur. first move I have left out all the reasoning and logic that make the idea of a BitLicense a farce on stilts. You can read my pieces on . From a pure perspective of innovation being good for society and the State’s inability to protect anyone, or predict the future, any regulation of Bitcoin is a huge mistake. Light bulbs are not like Bitcoins. Bitcoin has no mass, needs no factories and do not need to be transported. They are . The people who run businesses managing that data can be anywhere, and move there instantly. They do not have to close down factories, and relocate them. They don’t even have to move computers; all they need to do is buy hosting in another, better jurisdiction. And of course, only a would start a new Bitcoin business in New York. other that subject data fool The world is changing. Business as usual is off of the table, and if jurisdictions want innovation, investment and entrepreneurs in their territories, they are going to have to adapt to this new reality, rather than fight it. You can’t fight the future. A box of clear 100w incandescent bulbs, Edison fixture ↴

The Eureka Killers

Thank you for having elevated our evergreen community, we cannot wait to hear your story 💚 Introduction Introduction My I.T. superhero origin story goes like this: I got my degree in Theater and promptly realized the world didn’t need another short young scrawny nebbishy looking actor (at the time, Matthew Broderick had that corner of the market sewn up). I fell into IT training almost by accident, but it allowed me to feed my family while also having a very perform-ative element to it. I was able to take complex technical concepts and make them understandable, relatable, and even funny. After 5 years of in-class work I was ready to take my show on the road. I got a job at a “real” company and worked my way up the IT food chain from helpdesk to desktop support to sysadmin and network engineering. Then I took a weird sideways shift to work on “Tivoli”, a monitoring, inventory, and software distribution tool. From that point on, for the next 27 years, monitoring and observability was my jam. I used just about every tool out there, and applied it in environments that ranged from modest to moderate to mind-boggling. And then, 11 years ago, one of those monitoring vendors asked if I’d like to be one of their “Head Geeks”, a term that seemed cooler than “Technical Evangelist” - the prevailing term at the time. The role allowed me to leverage the technical knowledge I’d acquired up until then, but weave in my penchant for writing, public speaking, and being an all-around show-off. And that’s what I’ve been doing since. How Did You Start Writing? I’ve always been a person-who-writes. Whether it was diary entries or compulsive classroom notes (not about the subject. That was boring. But if you wanted to know who’s chair made a fart noise in social studies, I was your guy.) When I became a Head Geek / Developer Relations Advocate / Technical Evangelist / Dancing Bear the writing I did naturally became part of my job. How Has Technology Impacted The Way You Write? While I imagine myself as the type of writer who could do well with a trusty underwood or even a fancy IBM selectric, the truth is that there are a number of tools that I use in small but significant ways. Obviously the internet itself - with the ability to research almost anything without leaving the comfort of my chair - is a biggie. Likewise are utilities that have become so ubiquitous as to be virtually invisible at this point - everything from spell- and grammar-checking to word counters and beyond. Beyond that, the most important tool I have isn’t one that helps me do this, but rather a tool that helps me NOT do things: focus mode. My cell phone is another important tool (not to writing specifically, but to my life in general), but it’s also a huge stumbling block to me when I need to get work done. As a 57 year old guy with undiagnosed ADHD, someone who learned to cope back before all the amazing adaptations we have today, I have nevertheless amassed my own collection of coping mechanisms. Focus mode on my phone is one of them, and a big one at that. However, at the very top of my list of technologies that impact my writing are keyboards. Yes, I’m one of THOSE people. The kind of person who knows the difference between Cherry MX red, brown, or blue and have an opinion about it to boot. My all time favorite keyboard is an old Sun Microsystems Type 5 keyboard. It’s got just the right amount of play but also a softness to it. It also has a Mini-Din 8a connector. Yes, it looks like an old PS2 connector but it is decidedly NOT. Which is why I was DELIGHTED to drop $65 for an adapter that lets me use my favorite keyboard with modern equipment. an adapter For me, having a keyboard that feels good using is a big part of getting past my lizard brain that comes up with all kinds of reasons NOT to write. Share About Your Journey Highlights? (Share with us the highs and the lows, pivotal moments that reshaped your creative path.) The most important moment that helped me pivot from a-person-who-likes-to-write to “a writer” came 11 years ago, when my career shifted from being an individual contributor (specifically a IT practitioner who helped design, implement, and maintain monitoring systems) to a developer relations advocate for a software vendor. That role is ALL ABOUT writing, and it gave me the reason, time, and motivation to spend serious quality time at the keyboard, and make the mistakes necessary to learn and grow in the craft. What’s Your Creative Process? (Do you have a specific writing routine? How do you get ideas for your articles?) In her now-famous TED talk, “Your Elusive Creative Genius“, author Elizabeth Gilbert talked about her creative process. Rather than the cliched lightning stroke of inspiration, of the artist crying “Eureka” and leaping out of bed to immediately (and instantly, in one sitting) create a masterpiece, Ms. Gilbert said of herself, TED talk, “Your Elusive Creative Genius“ I’m a mule, and the way that I have to work is I have to get up at the same time every day, and sweat and labor and barrel through it really awkwardly. I’m a mule, and the way that I have to work is I have to get up at the same time every day, and sweat and labor and barrel through it really awkwardly. That is a really important point to share here, to anyone reading this and wondering “Why is this so hard for me? Maybe I’m just not a creative person.” Please never doubt that you, dear reader, ARE. It is hard for all of us. It is a slog, sometimes. It is work  - “…the dogsbody work of writing” was how another writer put it once. dogsbody That said, my creative process is a blend. There are days when the topic isn’t grabbing me and  the ideas simply aren’t there. In those moments, the advice of every writing coach on the planet holds true: keep writing anyway. Write anything. Write about not having any good ideas. Write about what you see out your window. Just. Write. But… Other moments - happily for me, many moments - the ideas fly faster than my fingers can type them. In those moments my ADHD brain isn’t a freight train, it’s the Broadway production of Starlight Express - dozens of actors on rollerblades streaking in multiple directions at breakneck speed, all vying for my attention. I bounce from typing a paragraph for the work at hand to jotting notes in Joplin - sometimes on the same topic, sometimes something completely disassociated. By the time I’m done, I might have 3 or 4 paragraphs of good material, and 6 or 7 other essay ideas (not all of them good). But if I try to focus down to one thing, I end up with no-thing. The act of saying “stop” to any of those ideas ends gives me a case of creative constipation. So my process has adapted to just keep the movement going, and clean up the mess afterward. Also… ew. Sorry for that mental image. Your Favorite Memory/Article(s) to This Day? Over the 11 years as a DevRel Advocate, I’ve had a chance to create a lot of stuff. With that volume of work, it’s hard to pick a favorite. Also, like coding itself, when I look back on older things I’ve written there are parts I like and there are parts that I wish I’d done better (but I also recognize I couldn’t have done them better because I wasn’t at that point in my journey yet.) create a lot of stuff With all of that said, so far this year, the piece I’ve enjoyed most is “HOW TO DEVREL: It Might Not Be New, But It’s Yours” HOW TO DEVREL: It Might Not Be New, But It’s Yours How Did You Hear About HackerNoon? Share With Us About Your Experience With HackerNoon. I originally heard about HackerNoon from one of my DevRel coworkers. I was looking for a community that had a large, vibrant, engaged audience where I could share my thoughts - ostensibly about tech, but also about whatever was top of mind for me - and I realized that HackerNoon, from the bright 8-bit aesthetic to the cheeky notifications, was the right place for my work. What Have You Learnt From Your Journey? I’ve been around the block a time or 10 so there are SO MANY things I’ve learned along the way: The single greatest skill anyone can have is the ability to quickly and comfortably say “I don’t know, but I would love to find out and share it with you.”
You will always learn more from your mistakes, mis-steps, and missed shots than you will from your nothing-but-net wins. Learning to see those moments for what they are and make the most of them is huge.
Just because you don’t feel like it doesn’t mean you shouldn’t do it. Conversely, just because you want to do it doesn’t mean you should.
Putting gas in the tank - yes I mean taking a break, but also reading, talking to other people, and experiencing other people’s creative work - is critical to your growth as a creator.
Anybody who tells you that you’re not enough - not old enough or young enough or experienced enough or skilled enough or whatever - is scared of you. Not just who you are today, but what you might become tomorrow if they don’t shut you down now. See them for what they are, and get past them as quickly as you can.

That also applies to the voice in your own head telling you that you can’t.
Yes, I have one of those voices too. I know how hard it is to shut it down or ignore it. Keep trying. The single greatest skill anyone can have is the ability to quickly and comfortably say “I don’t know, but I would love to find out and share it with you.” You will always learn more from your mistakes, mis-steps, and missed shots than you will from your nothing-but-net wins. Learning to see those moments for what they are and make the most of them is huge. Just because you don’t feel like it doesn’t mean you shouldn’t do it. Conversely, just because you want to do it doesn’t mean you should. Putting gas in the tank - yes I mean taking a break, but also reading, talking to other people, and experiencing other people’s creative work - is critical to your growth as a creator. Anybody who tells you that you’re not enough - not old enough or young enough or experienced enough or skilled enough or whatever - is scared of you. Not just who you are today, but what you might become tomorrow if they don’t shut you down now. See them for what they are, and get past them as quickly as you can.

That also applies to the voice in your own head telling you that you can’t.
Yes, I have one of those voices too. I know how hard it is to shut it down or ignore it. Keep trying. That also applies to the voice in your own head telling you that you can’t.
Yes, I have one of those voices too. I know how hard it is to shut it down or ignore it. Keep trying. That also applies to the voice in your own head telling you that you can’t. Yes, I have one of those voices too. I know how hard it is to shut it down or ignore it. Keep trying.

Meet HackerNoon Top Writer: Leon Adato - Discussing Transition to Tech, Writing, And Overall Journey

A Single Point of Access or Chaos — Unlocking Non-Blocking API Gateways

How ICPL Enhances Reward Function Efficiency and Tackles Complex RL Tasks

Table of Links Abstract and Introduction
Related Work
Problem Definition
Method
Experiments
Conclusion and References A. Appendix A.1. Full Prompts and A.2 ICPL Details A. 3 Baseline Details A.4 Environment Details A.5 Proxy Human Preference A.6 Human-in-the-Loop Preference 2 RELATED WORK Reward Design. In reinforcement learning, reward design is a core challenge, as rewards must both represent a desired set of behaviors and provide enough signal for learning. The most common approach to reward design is handcrafting, which requires a large number of trials by experts (Sutton, 2018; Singh et al., 2009). Since hand-coded reward design requires extensive engineering effort, several prior works have studied modeling the reward function with precollected data. For example, Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstration data (Arora & Doshi, 2021; Ng et al., 2000). With advances in pretrained foundation models, some recent works have also studied using large language models or vision-language models to provide reward signals (Ma et al., 2022; Fan et al., 2022; Du et al., 2023; Karamcheti et al., 2023; Kwon et al., 2023; Wang et al., 2024; Ma et al., 2024). Among these approaches, EUREKA (Ma et al., 2023) is the closest to our work, instructing the LLM to generate and select novel reward functions based on environment feedback with an evolutionary framework. However, as opposed to performing preference learning, EUREKA uses the LLM to design dense rewards that help with learning a hand-designed sparse reward. In contrast, ICPL focuses on finding a reward that correctly orders a set of preferences. However, we note that EUREKA also has a small, preliminary investigation combining human preferences with an LLM to generate human-preferred behaviors in a single scenario. We note that this experiment had the human provide their feedback in text, whereas we only use preference queries. This paper is a significantly scaled-up version of that investigation as well as a methodological study of how best to incorporate prior rounds of feedback. Human-in-the-loop Reinforcement Learning. Feedback from humans has been proven to be effective in training reinforcement learning agents that better match human preferences (Retzlaff et al., 2024; Mosqueira-Rey et al., 2023; Kwon et al., 2023). Previous works have investigated human feedback in various forms, such as trajectory comparisons, preferences, demonstrations, and corrections (Wirth et al., 2017; Ng et al., 2000; Jeon et al., 2020; Peng et al., 2024). Among these various methods, preference-based RL has been successfully scaled to train large foundation models for hard tasks like dialogue, e.g. ChatGPT (Ouyang et al., 2022). In LLM-based applications, prompting is a simple way to provide human feedback in order to align LLMs with human preferences (Giray, 2023; White et al., 2023; Chen et al., 2023). Iteratively refining the prompts with feedback from the environment or human users has shown promise in improving the output of the LLM (Wu et al., 2021; Nasiriany et al., 2024). This work extends the usage of the ability to control LLM behavior via in-context prompts. We aim to utilize interactive rounds of preference feedback between the LLM and humans to guide the LLM to generate reward functions that can elicit behaviors that align with human preferences. Authors:
(1) Chao Yu, Tsinghua University;
(2) Hong Lu, Tsinghua University;
(3) Jiaxuan Gao, Tsinghua University;
(4) Qixin Tan, Tsinghua University;
(5) Xinting Yang, Tsinghua University;
(6) Yu Wang, with equal advising from Tsinghua University;
(7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute;
(8) Eugene Vinitsky, with equal advising from New York University (zoeyuchao@gmail.com). This paper is available on arxiv under CC 4.0 license. Table of Links Abstract and Introduction Related Work Problem Definition Method Experiments Conclusion and References Abstract and Introduction Abstract and Introduction Related Work Related Work Problem Definition Problem Definition Method Method Experiments Experiments Conclusion and References Conclusion and References A. Appendix A. Appendix A.1. Full Prompts and A.2 ICPL Details A.1. Full Prompts and A.2 ICPL Details A. 3 Baseline Details A. 3 Baseline Details A.4 Environment Details A.4 Environment Details A.5 Proxy Human Preference A.5 Proxy Human Preference A.6 Human-in-the-Loop Preference A.6 Human-in-the-Loop Preference 2 RELATED WORK Reward Design . In reinforcement learning, reward design is a core challenge, as rewards must both represent a desired set of behaviors and provide enough signal for learning. The most common approach to reward design is handcrafting, which requires a large number of trials by experts (Sutton, 2018; Singh et al., 2009). Since hand-coded reward design requires extensive engineering effort, several prior works have studied modeling the reward function with precollected data. For example, Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstration data (Arora & Doshi, 2021; Ng et al., 2000). With advances in pretrained foundation models, some recent works have also studied using large language models or vision-language models to provide reward signals (Ma et al., 2022; Fan et al., 2022; Du et al., 2023; Karamcheti et al., 2023; Kwon et al., 2023; Wang et al., 2024; Ma et al., 2024). Reward Design Among these approaches, EUREKA (Ma et al., 2023) is the closest to our work, instructing the LLM to generate and select novel reward functions based on environment feedback with an evolutionary framework. However, as opposed to performing preference learning, EUREKA uses the LLM to design dense rewards that help with learning a hand-designed sparse reward. In contrast, ICPL focuses on finding a reward that correctly orders a set of preferences. However, we note that EUREKA also has a small, preliminary investigation combining human preferences with an LLM to generate human-preferred behaviors in a single scenario. We note that this experiment had the human provide their feedback in text, whereas we only use preference queries. This paper is a significantly scaled-up version of that investigation as well as a methodological study of how best to incorporate prior rounds of feedback. Human-in-the-loop Reinforcement Learning. Feedback from humans has been proven to be effective in training reinforcement learning agents that better match human preferences (Retzlaff et al., 2024; Mosqueira-Rey et al., 2023; Kwon et al., 2023). Previous works have investigated human feedback in various forms, such as trajectory comparisons, preferences, demonstrations, and corrections (Wirth et al., 2017; Ng et al., 2000; Jeon et al., 2020; Peng et al., 2024). Among these various methods, preference-based RL has been successfully scaled to train large foundation models for hard tasks like dialogue, e.g. ChatGPT (Ouyang et al., 2022). In LLM-based applications, prompting is a simple way to provide human feedback in order to align LLMs with human preferences (Giray, 2023; White et al., 2023; Chen et al., 2023). Iteratively refining the prompts with feedback from the environment or human users has shown promise in improving the output of the LLM (Wu et al., 2021; Nasiriany et al., 2024). This work extends the usage of the ability to control LLM behavior via in-context prompts. We aim to utilize interactive rounds of preference feedback between the LLM and humans to guide the LLM to generate reward functions that can elicit behaviors that align with human preferences. Human-in-the-loop Reinforcement Learning. Authors: (1) Chao Yu, Tsinghua University; (2) Hong Lu, Tsinghua University; (3) Jiaxuan Gao, Tsinghua University; (4) Qixin Tan, Tsinghua University; (5) Xinting Yang, Tsinghua University; (6) Yu Wang, with equal advising from Tsinghua University; (7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute; (8) Eugene Vinitsky, with equal advising from New York University (zoeyuchao@gmail.com). Authors: Authors: (1) Chao Yu, Tsinghua University; (2) Hong Lu, Tsinghua University; (3) Jiaxuan Gao, Tsinghua University; (4) Qixin Tan, Tsinghua University; (5) Xinting Yang, Tsinghua University; (6) Yu Wang, with equal advising from Tsinghua University; (7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute; (8) Eugene Vinitsky, with equal advising from New York University ( zoeyuchao@gmail.com ). zoeyuchao@gmail.com This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Hacking Reinforcement Learning with a Little Help from Humans (and LLMs)

Authors:
(1) Chao Yu, Tsinghua University;
(2) Hong Lu, Tsinghua University;
(3) Jiaxuan Gao, Tsinghua University;
(4) Qixin Tan, Tsinghua University;
(5) Xinting Yang, Tsinghua University;
(6) Yu Wang, with equal advising from Tsinghua University;
(7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute;
(8) Eugene Vinitsky, with equal advising from New York University (zoeyuchao@gmail.com). Table of Links Abstract and Introduction
Related Work
Problem Definition
Method
Experiments
Conclusion and References A. Appendix A.1. Full Prompts and A.2 ICPL Details A. 3 Baseline Details A.4 Environment Details A.5 Proxy Human Preference A.6 Human-in-the-Loop Preference ABSTRACT Designing reward functions is a core component of reinforcement learning but can be challenging for truly complex behavior. Reinforcement Learning from Human Feedback (RLHF) has been used to alleviate this challenge by replacing a hand-coded reward function with a reward function learned from preferences. However, it can be exceedingly inefficient to learn these rewards as they are often learned tabula rasa. We investigate whether Large Language Models (LLMs) can reduce this query inefficiency by converting an iterative series of human preferences into code representing the rewards. We propose In-Context Preference Learning (ICPL), a method that uses the grounding of an LLM to accelerate learning reward functions from preferences. ICPL takes the environment context and task description, synthesizes a set of reward functions, and then repeatedly updates the reward functions using human rankings of videos of the resultant policies. Using synthetic preferences, we demonstrate that ICPL is orders of magnitude more efficient than RLHF and is even competitive with methods that use ground-truth reward functions instead of preferences. Finally, we perform a series of human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop. Additional information and videos are provided at https://sites.google.com/view/few-shot-icpl/home. 1 INTRODUCTION Designing state-of-the-art agents using reinforcement learning (RL) often requires the design of reward functions that specify desired and undesirable behaviors. However, for sufficiently complex tasks, designing an effective reward function remains a significant challenge. This process is often difficult, and poorly designed rewards can lead to biased, misguided, or even unexpected behaviors in RL agents (Booth et al., 2023). Recent advances in large language models (LLMs) have shown potential in tackling this challenge as they are able to zero-shot generate reward functions that satisfy a task specification (Yu et al., 2023). However, the ability of LLMs to directly write a reward function is limited when task complexity increases or tasks are out-of-distribution from pre-training data. As an additional challenge, humans may be unable to perfectly specify their desired agent behavior in text. Human-in-the-loop learning (HL) offers a potential enhancement to the reward design process by embedding human feedback directly into the learning process. A ubiquitous approach is preferencebased RL where preferences between different agent behaviors serves as the primary learning signal. Instead of relying on predefined rewards, the agent learns a reward function aligned with human preferences. This interactive approach has shown success in various RL tasks, including standard benchmarks (Christiano et al., 2017; Ibarz et al., 2018), encouraging novel behaviors (Liu et al., 2020; Wu et al., 2021), and overcoming reward exploitation (Lee et al., 2021a). However, in more complex tasks involving extensive agent-environment interactions, preference-based RL often demands hundreds or thousands of human queries to provide effective feedback. For instance, a robotic arm button-pushing task requires over 10k queries to learn reasonable behavior (Lee et al.), which could be a major bottleneck. In this work, we introduce a novel method, In-Context Preference Learning (ICPL), which significantly enhances the sample efficiency of preference learning through LLM guidance. Our primary insight is that we can harness the coding capabilities of LLMs to autonomously generate reward functions, then utilize human preferences through in-context learning to refine these functions. Specifically, ICPL leverages an LLM, such as GPT-4, to generate executable, diverse reward functions based on the task description and environment source code. We acquire preferences by evaluating the agent behaviors resulting from these reward functions, selecting the most and least preferred behaviors. The selected functions, along with historical data such as reward traces of the generated reward functions from RL training, are then fed back into the LLM to guide subsequent iterations of reward function generation. We hypothesize that as a result of its grounding in text data, ICPL will be able to improve the quality of the reward function through incorporating the preferences and the history of the generated reward functions, ensuring they align more and more closely with human preferences. Unlike evolutionary search methods like EUREKA Ma et al. (2023), there is no ground-truth reward function that the LLM can use to evaluate agent performance, and thus, success here would demonstrate that LLMs have some native preference-learning capabilities. To study the effectiveness of ICPL, we perform experiments on a diverse set of RL tasks. For scalability, we first study tasks with synthetic preferences where a ground-truth reward function is used to assign preference labels. We observe that compared to traditional preference-based RL algorithms, ICPL achieves over a 30 times reduction in the required number of preference queries to achieve equivalent or superior performance. Moreover, ICPL attains performance comparable to, or even better than, methods that have access to a ground truth specification of the reward function Ma et al. (2023). Finally, we test ICPL on a particularly challenging task, “making a humanoid jump like a real human,” where designing a reward is difficult. By using real human feedback, our method successfully trained an agent capable of bending both legs and performing stable, human-like jumps, showcasing the potential of ICPL in tasks where human intuition plays a critical role. In summary, the contributions of the paper are the following: • We propose ICPL, an LLM-based preference learning algorithm. Over a synthetic set of preferences, we demonstrate that ICPL can iteratively output rewards that increasingly reflect preferences. Via a set of ablations, we demonstrate that this improvement is on average monotonic, suggesting that preference learning is occurring as opposed to a random search. • We demonstrate, via human-in-the-loop trials, that ICPL is able to work effectively with humans-in-the-loop despite significantly noisier preference labels. • We demonstrate that ICPL sharply outperforms tabula-rasa RLHF methods and is also competitive with methods that rely on access to a ground-truth reward. This paper is available on arxiv under CC 4.0 license. Authors: (1) Chao Yu, Tsinghua University; (2) Hong Lu, Tsinghua University; (3) Jiaxuan Gao, Tsinghua University; (4) Qixin Tan, Tsinghua University; (5) Xinting Yang, Tsinghua University; (6) Yu Wang, with equal advising from Tsinghua University; (7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute; (8) Eugene Vinitsky, with equal advising from New York University (zoeyuchao@gmail.com). Authors: Authors: (1) Chao Yu, Tsinghua University; (2) Hong Lu, Tsinghua University; (3) Jiaxuan Gao, Tsinghua University; (4) Qixin Tan, Tsinghua University; (5) Xinting Yang, Tsinghua University; (6) Yu Wang, with equal advising from Tsinghua University; (7) Yi Wu, with equal advising from Tsinghua University and the Shanghai Qi Zhi Institute; (8) Eugene Vinitsky, with equal advising from New York University ( zoeyuchao@gmail.com ). zoeyuchao@gmail.com Table of Links Abstract and Introduction Related Work Problem Definition Method Experiments Conclusion and References Abstract and Introduction Abstract and Introduction Related Work Related Work Problem Definition Problem Definition Method Method Experiments Experiments Conclusion and References Conclusion and References A. Appendix A. Appendix A.1. Full Prompts and A.2 ICPL Details A.1. Full Prompts and A.2 ICPL Details A. 3 Baseline Details A. 3 Baseline Details A.4 Environment Details A.4 Environment Details A.5 Proxy Human Preference A.5 Proxy Human Preference A.6 Human-in-the-Loop Preference A.6 Human-in-the-Loop Preference ABSTRACT Designing reward functions is a core component of reinforcement learning but can be challenging for truly complex behavior. Reinforcement Learning from Human Feedback (RLHF) has been used to alleviate this challenge by replacing a hand-coded reward function with a reward function learned from preferences. However, it can be exceedingly inefficient to learn these rewards as they are often learned tabula rasa. We investigate whether Large Language Models (LLMs) can reduce this query inefficiency by converting an iterative series of human preferences into code representing the rewards. We propose In-Context Preference Learning (ICPL), a method that uses the grounding of an LLM to accelerate learning reward functions from preferences. ICPL takes the environment context and task description, synthesizes a set of reward functions, and then repeatedly updates the reward functions using human rankings of videos of the resultant policies. Using synthetic preferences, we demonstrate that ICPL is orders of magnitude more efficient than RLHF and is even competitive with methods that use ground-truth reward functions instead of preferences. Finally, we perform a series of human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop. Additional information and videos are provided at https://sites.google.com/view/few-shot-icpl/home. 1 INTRODUCTION Designing state-of-the-art agents using reinforcement learning (RL) often requires the design of reward functions that specify desired and undesirable behaviors. However, for sufficiently complex tasks, designing an effective reward function remains a significant challenge. This process is often difficult, and poorly designed rewards can lead to biased, misguided, or even unexpected behaviors in RL agents (Booth et al., 2023). Recent advances in large language models (LLMs) have shown potential in tackling this challenge as they are able to zero-shot generate reward functions that satisfy a task specification (Yu et al., 2023). However, the ability of LLMs to directly write a reward function is limited when task complexity increases or tasks are out-of-distribution from pre-training data. As an additional challenge, humans may be unable to perfectly specify their desired agent behavior in text. Human-in-the-loop learning (HL) offers a potential enhancement to the reward design process by embedding human feedback directly into the learning process. A ubiquitous approach is preferencebased RL where preferences between different agent behaviors serves as the primary learning signal. Instead of relying on predefined rewards, the agent learns a reward function aligned with human preferences. This interactive approach has shown success in various RL tasks, including standard benchmarks (Christiano et al., 2017; Ibarz et al., 2018), encouraging novel behaviors (Liu et al., 2020; Wu et al., 2021), and overcoming reward exploitation (Lee et al., 2021a). However, in more complex tasks involving extensive agent-environment interactions, preference-based RL often demands hundreds or thousands of human queries to provide effective feedback. For instance, a robotic arm button-pushing task requires over 10k queries to learn reasonable behavior (Lee et al.), which could be a major bottleneck. In this work, we introduce a novel method, In-Context Preference Learning (ICPL), which significantly enhances the sample efficiency of preference learning through LLM guidance. Our primary insight is that we can harness the coding capabilities of LLMs to autonomously generate reward functions, then utilize human preferences through in-context learning to refine these functions. Specifically, ICPL leverages an LLM, such as GPT-4, to generate executable, diverse reward functions based on the task description and environment source code. We acquire preferences by evaluating the agent behaviors resulting from these reward functions, selecting the most and least preferred behaviors. The selected functions, along with historical data such as reward traces of the generated reward functions from RL training, are then fed back into the LLM to guide subsequent iterations of reward function generation. We hypothesize that as a result of its grounding in text data, ICPL will be able to improve the quality of the reward function through incorporating the preferences and the history of the generated reward functions, ensuring they align more and more closely with human preferences. Unlike evolutionary search methods like EUREKA Ma et al. (2023), there is no ground-truth reward function that the LLM can use to evaluate agent performance, and thus, success here would demonstrate that LLMs have some native preference-learning capabilities. To study the effectiveness of ICPL, we perform experiments on a diverse set of RL tasks. For scalability, we first study tasks with synthetic preferences where a ground-truth reward function is used to assign preference labels. We observe that compared to traditional preference-based RL algorithms, ICPL achieves over a 30 times reduction in the required number of preference queries to achieve equivalent or superior performance. Moreover, ICPL attains performance comparable to, or even better than, methods that have access to a ground truth specification of the reward function Ma et al. (2023). Finally, we test ICPL on a particularly challenging task, “making a humanoid jump like a real human,” where designing a reward is difficult. By using real human feedback, our method successfully trained an agent capable of bending both legs and performing stable, human-like jumps, showcasing the potential of ICPL in tasks where human intuition plays a critical role. In summary, the contributions of the paper are the following: • We propose ICPL, an LLM-based preference learning algorithm. Over a synthetic set of preferences, we demonstrate that ICPL can iteratively output rewards that increasingly reflect preferences. Via a set of ablations, we demonstrate that this improvement is on average monotonic, suggesting that preference learning is occurring as opposed to a random search. • We demonstrate, via human-in-the-loop trials, that ICPL is able to work effectively with humans-in-the-loop despite significantly noisier preference labels. • We demonstrate that ICPL sharply outperforms tabula-rasa RLHF methods and is also competitive with methods that rely on access to a ground-truth reward. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Stories, News & Mentions about EVREKA

Categories

Trending Topics

Stories, News & Mentions about EVREKA

Evreka’s Stories on HackerNoon

Evreka's News & Mentions

Categories

Trending Topics