andreer

[andreer]

“Hello World” in morse: turning an FPGA into a radio transmitter

I’ll try here to explain everything that is going on in this picture, but first, the origin story. I recently tweeted this photograph of my first ever FPGA project:

Actually this is from a slightly different angle. Bonus! I got more attention on twitter from this than I’d ever gotten before, with lots of notifications on my phone throughout the next days, and of course I’m super flattered. So when a few people request a better explanation I can’t say no!

Even though the people who responded to this are probably a highly technical audience, I’ll try to start very basic and include a lot of links, for the benefit(?) of my other acquaintances whom I’ll pester to read this - so feel free to skim ahead if you’d like.

What is an FPGA?

First, what is this FPGA thing? You can see one in my photo, in front of my keyboard, connected to my computer by a white USB cable. Although this one is small, it contains a lot of the basic components from which digital circuits (like a CPU) are built: logic gates which can compute things, and flip-flops and RAM which can store stuff.

But unlike a CPU, the components in an FPGA are reconfigurable: you get to decide how they are connected together. And you can do it without having to solder millions of tiny wires and losing tiny transistors between the keys in your keyboard. Just describe how you’d like the circuit to be connected on your PC, using a “drag-and-drop and draw wires” program or a text-based “hardware description language”, and through the magic of logic synthesis you can make your circuit come alive within the FPGA.

The trick to it is that all the components in the FPGA are already all connected together in lots of ways, and your computer is just telling the FPGA which of these connections should be on or off - like a light switch decides if your lamp should be connected to power or not.

There are also connections from inside the FPGA to the outside of the board, where you can connect external components like an LED, an LCD display or a keyboard to your circuit.

The specific FPGA I’m using is the TinyFPGA BX. It’s small, very cheap at $38, and is fully open source hardware. All the software you need is open source too, so it’s all free and easy to get access to, and the TinyFPGA page has lots of great links to everything you need to get started.

How did I turn it into a radio transmitter?

Here comes the plot twist: it was a radio transmitter all along.

In order to keep the digital logic of the FPGA in sync and working in unison, it is equipped with an oscillator which transmits a clock signal - a signal that switches between high and low at regular intervals. You’ve probably heard about “clock rates” before, because it is one of the things that decides the performance of a computer, and these days it is usually measured in GHz.

Now simply by directing the FPGA to route the signal from this clock source out to one of it’s external connections, and connecting it to a small piece of wire, we have a radio transmitter! See the screenshot of this setup in the Icestudio FPGA programming environment, possibly the simplest example imaginable:

The output, and the wire connected to it, will be driven to high and low voltage levels by the clock signal, and waves of electromagnetism will spread out across the universe. Imagine the piece of wire (or “antenna”) as a tiny oar splashing up and down in the fabric of space time, in rythm with the clock signal.

But there are some serious disadvantages to the radio signal we are generating here. First, the “oar” of the antenna is very small, and the “muscle” of the clock signal is not very strong, so it’s can’t make very big waves - the transmission is not very powerful. But that is probably for the best, because trying to transmit using a square wave like the 16MHz clock signal will, in addition to a signal on 16MHz, create a lot of “harmonics” - unwanted signals on other frequencies.

Because I’m an amateur radio enthusiast, I have a very nice radio receiver sitting on my desk that can receive and display the transmitted signal in a very nice way (specifically a LimeSDR, It’s visible on the right in my first picture). Here’s a screenshot from GQRX, the program I’m using to control it:

It’s tuned to 16 MHz (the red vertical line), but you can also see some more of the radio spectrum on each side of it. In addition to the sharp peak of the main signal on 16MHz, there is a very apparent cloud of noise surrounding it - which we can see here as a raised area on the "panadapter” up top, or a brighter, yellow color around the 16MHz line on the “waterfall” below. There are also weaker, but still very visible symmetrical peaks to the left and right of the main signal.

Outside of what we can see in this screenshot, there’s also strong “harmonics” being transmitted on other frequencies, like 8 or 24 MHz. In short, what we’re doing here is generating what’s usually called EMI - ElectroMagnetic Interference, and it’s a Bad Thing™. This is the reason real products need conductive shielding, and try to keep wires as short as possible to keep them from acting as antennas. And of course, you should never connect a signal like this to an amplifier - because you don’t want to end up disturbing someone’s pacemaker or interrupting emergency communications.

(Just to make sure noone gets the wrong idea, the TinyFPGA BX itself seems well shielded and I’m hardly able to detect it being on without attaching the “antenna”.)

What you’d want, and what “real” radio transmitters generate, is something closer to a nice smooth sine wave like this:

Now it’s quite possible to get from a square wave to this, by passing the signal through a “filter” which can be easily constructed from inductors and capacitators. Well, if you consider soldering tiny components easy ...

I was able to assemble a kit for such a filter, allowing me to transmit “real” radio signals from the output pins of a Raspberry Pi with no additional components, and be received across town several kilometers away! And although that was with a much better antenna, it was without any amplification. So again, a warning about being very careful about what you transmit, intentionally or unintentionally, because even very weak signals can be heard quite far away.

Now there’s a couple of more things that are Bad™ about the 16MHz square wave signal we generated earlier: First, it’s not a frequency on which I am legally allowed to transmit(!). The radio spectrum is quite heavily regulated, but there are some frequency ranges for “free” use. These are typically quite crowded. There is one area around 433 MHz  that is used heavily by all sorts of gadgets like wireless temperature monitors or doorbells, and another around 2.4GHz which is very heavily used by Wi-Fi, Bluetooth and other high-bandwidth devices.

The other problem with our original signal is that it is very boring - just a steady buzz at 16MHz, hardly worth transmitting or listening to.

As a licensed (though inexperienced) amateur radio operator (my callsign in LB7HH!), I also have permission to transmit on amateur radio bands, with certain requirements. One of them is that during any transmission, I have to regularly include my call sign.

So with some copy-pasting and slight tweaks, here’s my first FPGA project:

The large rectangle on the left is a block of Verilog hardware description code - it’s a slight modification of the code from the TinyFPGA-BX project template. The code in the template blinks the LED in a SOS morse pattern. By doubling the size of the string, I was just exactly able to fit my callsign in morse code instead!

To the right, I first use the builtin “PrescalerN” component which can reduce the clock frequency by dividing it with 2^N. Lucky for me, there is a value of N that results in a frequency I’m legally allowed to transmit on - 2MHz, which is exactly the upper edge of the 160M band.

The last component is an AND gate which combines the two signals, only letting the 2MHz frequency signal through when the morse signal is high. And then it’s connected out to PIN_2 and my “antenna”. And that’s all there is to it.

So how powerful is this transmitter? I haven’t done the math, but judging by what similar hacks can do it should be around milliWatt at most, and with a very poor antenna the effective radiated power is much lower still. Which is actually a good thing, as I didn’t build that filter for 2MHz yet.

And I probably won’t, because the project I got this FPGA for is actually very different - implementing my own CPU design. I’ll just have to learn a lot more Verilog first ...

#fpga #verilog #radio #ham radio #transmitter #rfi #emi #morse

[andreer]

Open Sourcing Vespa, Yahoo’s Big Data Processing and Serving Engine

By Jon Bratseth, Distinguished Architect, Vespa

Ever since we open sourced Hadoop in 2006, Yahoo – and now, Oath – has been committed to opening up its big data infrastructure to the larger developer community. Today, we are taking another major step in this direction by making Vespa, Yahoo’s big data processing and serving engine, available as open source on GitHub.

Building applications increasingly means dealing with huge amounts of data. While developers can use the Hadoop stack to store and batch process big data, and Storm to stream-process data, these technologies do not help with serving results to end users. Serving is challenging at large scale, especially when it is necessary to make computations quickly over data while a user is waiting, as with applications that feature search, recommendation, and personalization.

By releasing Vespa, we are making it easy for anyone to build applications that can compute responses to user requests, over large datasets, at real time and at internet scale – capabilities that up until now, have been within reach of only a few large companies.

Serving often involves more than looking up items by ID or computing a few numbers from a model. Many applications need to compute over large datasets at serving time. Two well-known examples are search and recommendation. To deliver a search result or a list of recommended articles to a user, you need to find all the items matching the query, determine how good each item is for the particular request using a relevance/recommendation model, organize the matches to remove duplicates, add navigation aids, and then return a response to the user. As these computations depend on features of the request, such as the user’s query or interests, it won’t do to compute the result upfront. It must be done at serving time, and since a user is waiting, it has to be done fast. Combining speedy completion of the aforementioned operations with the ability to perform them over large amounts of data requires a lot of infrastructure – distributed algorithms, data distribution and management, efficient data structures and memory management, and more. This is what Vespa provides in a neatly-packaged and easy to use engine.

With over 1 billion users, we currently use Vespa across many different Oath brands – including Yahoo.com, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Gemini, Flickr, and others – to process and serve billions of daily requests over billions of documents while responding to search queries, making recommendations, and providing personalized content and advertisements, to name just a few use cases. In fact, Vespa processes and serves content and ads almost 90,000 times every second with latencies in the tens of milliseconds. On Flickr alone, Vespa performs keyword and image searches on the scale of a few hundred queries per second on tens of billions of images. Additionally, Vespa makes direct contributions to our company’s revenue stream by serving over 3 billion native ad requests per day via Yahoo Gemini, at a peak of 140k requests per second (per Oath internal data).

With Vespa, our teams build applications that:

Select content items using SQL-like queries and text search

Organize all matches to generate data-driven pages

Rank matches by handwritten or machine-learned relevance models

Serve results with response times in the low milliseconds

Write data in real-time, thousands of times per second per node

Grow, shrink, and re-configure clusters while serving and writing data

To achieve both speed and scale, Vespa distributes data and computation over many machines without any single master as a bottleneck. Where conventional applications work by pulling data into a stateless tier for processing, Vespa instead pushes computations to the data. This involves managing clusters of nodes with background redistribution of data in case of machine failures or the addition of new capacity, implementing distributed low latency query and processing algorithms, handling distributed data consistency, and a lot more. It’s a ton of hard work!

As the team behind Vespa, we have been working on developing search and serving capabilities ever since building alltheweb.com, which was later acquired by Yahoo. Over the last couple of years we have rewritten most of the engine from scratch to incorporate our experience onto a modern technology stack. Vespa is larger in scope and lines of code than any open source project we’ve ever released. Now that this has been battle-proven on Yahoo’s largest and most critical systems, we are pleased to release it to the world.

Vespa gives application developers the ability to feed data and models of any size to the serving system and make the final computations at request time. This often produces a better user experience at lower cost (for buying and running hardware) and complexity compared to pre-computing answers to requests. Furthermore it allows developers to work in a more interactive way where they navigate and interact with complex calculations in real time, rather than having to start offline jobs and check the results later.

Vespa can be run on premises or in the cloud. We provide both Docker images and rpm packages for Vespa, as well as guides for running them both on your own laptop or as an AWS cluster.

We’ll follow up this initial announcement with a series of posts on our blog showing how to build a real-world application with Vespa, but you can get started right now by following the getting started guide in our comprehensive documentation.

Managing distributed systems is not easy. We have worked hard to make it easy to develop and operate applications on Vespa so that you can focus on creating features that make use of the ability to compute over large datasets in real time, rather than the details of managing clusters and data. You should be able to get an application up and running in less than ten minutes by following the documentation.

We can’t wait to see what you’ll build with it!

[andreer]

Reblog this picture of me holding a Family Size box of Honey Nut Cheerios? I’d really appreciate it.

[andreer]

The Japanese Museum of Rocks That Look Like Faces

[andreer]

All aboard the hype train!

Nintendo Switch is coming March 2017!

[andreer]

слабо освещенные питание для всех!

[andreer]

[andreer]

På moskussafari!

[andreer]

Med elbil over Dovre! (ved Dombås, Norway)

[andreer]

Midnight Radio. Written by Ehud Lavski. Art by Yael Nathan. If you like it, please share.

Contact: [email protected]

[andreer]

Tear down this wall!

“We can’t work as efficiently as smaller companies. We’re just too big. We’ve got too many systems, too many customers and too many employees.”

If I had a dollar every time I heard something along those lines.  Too big. Bullshit. Your problem is not your size, whether measured by customers, systems or employees.  Your problem is how you manage them. Centralized control does not scale well.  That’s your problem. You’re trying to control all of your systems, all of your employees and all of your customers the same way.  That’s what’s slowing you down.  5 year plans and centralized control: this is software development communism.

Instead of letting go and delegating control to the various application teams, you obsess over creating a constellation of applications that looks nice on a diagram. A diagram that lets you understand the entirety of everything and how it’s all connected. One integration platform. One process engine.  Instead of every app having its own domain model, you create one common one for all to use.  Much easier for the person at the top to keep track of.  Worse for everyone else.

Simple example.  Say you have several applications that all need to generate, archive and distribute documents.  Today they all have their own separate code to do this. Duplication! The cries go out. We need to create a common document handling service they can all use.  Ok, NOW you have a problem. Because now you’ve got coupling.  And coupling is a much worse enemy than duplication.  The number of systems doesn’t matter nearly as much as the amount of coupling you have between them.  Instead of 3, say, separate, standalone apps that can be updated at will and tested in complete isolation, you now have 3 apps that are all connected to the same service. What if app 1 needs that service to behave a little bit differently in certain cases? To make this change you now need to retest your entire portfolio to ensure your new change didn’t break anything for the other connected apps.  Alternatively, you can make the change in the client app itself, thus allowing document handling logic to seep out of the document service and exist in both the common document service AND each individual application.  Over time, typically both things happen.  The common service grows and grows and gets more and more complex and configurable.  Each client app only uses a small fraction of the functionality provided.  It also gets harder and harder to maintain, as the intent of the code is hidden in configurabilty.  At the same time a substantial portion of document handling logic is added to each of the client apps to compensate for the functionality that has not been able to be added to the common module.  A complete mess.

If you don’t create a common service, you will most likely have a bit of duplication. This means that in the case where something needs to be updated – say the archiving system is upgraded - the required change will have to be implemented several times over. Every system using the archive must be updated.  This is annoying manual labor. But it is not difficult.  Writing and maintaining a common module that has to work for every possible client, requires less annoying manual grunt work, but it is very difficult.  Know what tradeoffs you’re making.

Sometimes common services pay off of course.  If they’re small, and only do one thing and do that same thing for all clients, they can be very valuable.  Which brings me to my second point.  Don’t worry about the connections looking messy when you draw every app and service in one diagram.  If there are 400 applications, and each one is connected to 3 other ones, it’s going to look appalling. Don’t draw that diagram! The only one interested in that diagram is the enterprise architect. Screw him (or her). The architect is far less important than your end user.  The purpose of your system is to provide meaning and value to the end user, not your enterprise architect.  Any one user will never be using all your systems. They will use a couple of them. Those are the diagrams you need to focus on. That’s where your energy should be spent. Focus on what the users need, and how the data they are providing or receiving is passed through the part of the system they are in contact with.  If that diagram is full of lots of systems, then you might have a problem. If for a user to enter some data, you need to go from a user interface app, to a process engine, to a document handling service, to a queue, to an archive and then back again.  That might be something you’d want to simplify.  This is the only context in which you should worry about the number of systems and the amount of coupling.  But reducing the amounts of systems for each user, will increase the number of systems at the enterprise level (and vice versa). This is why the enterprise architect doesn’t like it. We need to remind ourselves who we are really working to please. Our users or our architects?

Another argument I hear a lot in favor of introducing common systems, and reducing the number of applications is that “users dislike having so many systems to work with”. No they don’t! Ask anyone – what do you prefer: SAP (one system that does it all) or the iPhone with a multitude of apps downloaded from the App Store? They prefer the iPhone.  There are hundreds if not thousands of apps for every kind of need. But this is much better than the alternative: One app that serves everyone badly.  This idea that users dislike having many systems is bogus.  You know what they don’t like? They don’t like having to remember lots of usernames and passwords.  With a common authentication system (yes, I’m advocating reuse!) that problem is solved. What users don’t like is bad user experiences. It’s much easier to create good user experiences if you’re allowed to create several tailored small apps that serve very specific purposes.  If you only have one app, and it has to do everything imaginable for everyone, it’s not going to be very user friendly.  

So, no, I don’t accept your excuse.  Large companies are not inefficient, slow and bureaucratic because of their size. They’re inefficient, slow and bureaucratic because of their communism. Mr. Enterprise Architect; tear down this wall!

[andreer]

So, You Say You Like Cyberpunk?

As a result of my being on sabbatical since the end of October, I’ve had the ability to write my next novel, “Gifted: Downline.”

It’s currently in rough alpha form, and is being edited and redrafted in to a beta form that I will eventually be shopping to agents.

But, since so many of you seemed to like the alpha draft of “The Widow Wore Gold,” I figure I might as well offer it up here for an alpha read.

The draft you would be reading would not be the final, published draft. It’s a rough alpha, with a lot of problems and a lot of flaws. But it’s finished, and it’s 89 thousand words worth of Cyberpunk science fiction.

(For the record, while researching the science in the novel, my search history consisted of:

Stochastic Theory Determinism Quantum Entanglement Delivery Methods and Symptoms of Cyanide Compounds

And so much more!)

Anyway - if I get at least ten positive replies to this post, I’ll work up a posting schedule for the story.

Keep in mind, though: If you vote “yes” to reading the thing, I’ll be asking you to comment on it, reblog it, and generally spread it around. As I’ve said before: my readership is crowdsourced, and you are that crowd.

So:

Wanna read a story?

[andreer]

i put “All I Want for Christmas is You” through a MIDI converter, and then back through an mp3 converter

the result is this garbage

[andreer]

Manage Your Gmail Right From Yahoo Mail

By Shiv Shankar, Senior Product Manager

When we launched the redesigned Yahoo Mail app in October, users called it “slick,” “speedy” and “elegant.” The app now has a four-star rating in both the Apple App Store and Google Play. We introduced multiple mailbox management for Outlook.com, Hotmail and AOL Mail. Yet, something was missing from the list - the ability to connect your Gmail and Google Apps account to Yahoo Mail. You asked for it and now it’s here!

All the features you need and love in Yahoo Mail – powerful search, smart contacts, rich compose features and Yahoo Account Key for password-free sign-in – are now available with Gmail. Here’s your path to a more convenient, powerful and personal email experience. You can now search across your entire history of emails, from day one, for all connected mailboxes! (Not just your most recent emails, which is how it’s done in many apps today).

We even sync every single action across all your mailboxes, like moving emails between folders, deleting emails, marking them as read, etc. When you compose an email, you can send from any account you’ve connected (i.e. personal, school, work, etc.) even though you’re using Yahoo Mail.

Here are some of the Yahoo Mail features that you can now use across your Gmail emails:

Sign in, Password Free: With Yahoo Account Key on mobile, you no longer have to memorize complicated passwords. Account Key makes signing in to your Yahoo account as easy as tapping a button.

Compose Engaging Emails: Easily add photos and videos with the new Yahoo Mail app. Or, try the new compose feature on desktop to find and add not only photos and videos, but also GIFs, files and links too. Just click on the “+” sign when composing a new message.

Search Smarter: Find what you need quickly with our search results pages that not only display relevant messages but also files and photos from all your connected mailboxes.

Access Contacts Across Your Mailboxes: Our intelligent contacts manager, powered by Xobni technology, makes suggestions for who you should email based on your habits. Yahoo Mail also displays contact cards that include information you may need about someone (i.e. email addresses, phone numbers, social media info, email history and more).

Multiple mailbox support for Gmail within Yahoo Mail is available globally today in the Yahoo Mail app and on the desktop browser. To connect your Gmail account, go into your Account Settings in the Yahoo Mail mobile app or on the desktop.

We’ve also announced new features and capabilities for the Yahoo Mail iOS and Android apps.

Go on now and email your heart out, straight from Yahoo Mail, no matter which email service you use.

[andreer]

(via Saturday Morning Cartoons: Baopu #15) by Yao Xiao

words to remember

[andreer]

Someone should set this to music.

‘A wonderful meeting of cultures which can only fill you with hope in these trying and uncertain times…’

That’s how Gary tried to sell his own dinner to himself.

Just like the ceaselessly rude stream of so-called people he called up every day at work—he wasn’t buying.

No one wanted to change their stationery providers, literally no one in the whole of the country.

Yet still he was tasked with ringing around small businesses, holding out hope against hope that they didn’t have a business account with Staples.

They always did. They were always delighted to tell him so too.

It hadn’t meant to turn out like this. He’d been a Theatre student with the songs of Sondheim in his heart and a willingness to sleep with casting directors of every shape and size.

His big break never came, and eventually those telesales jobs he’d worked to pay the bills became his life—the annual squalid, regretful, Christmas party romance the highlight of every year.

There were still daydreams to be had of being wined and dined by a Lloyd-Webber or sharing a canape with Michael Crawford.

Yet they brought equal parts sorrow to joy, and worse, his imagination had been dulled by years of rejection in the worlds of both stagecraft and office supplies.

The only constant was stale naan.

Stale naan and beans.

Stale naan and beans and cheese.

[andreer]

#SoundCloud #music #Steven Universe #Adventure Time #Everything Stays #Stakes #Mini-Series