Deploying AI is the hardest part

I spent three days at the O’Reilly AI Conference in London a couple of weeks back. It was incredibly informative in terms of applying AI, understanding some very new techniques and in particular, deploying and running models.

Kubeflow came up time and again as a way to build pipelines – connecting various data processing, training, analysis and serving steps by wrapping them in Docker containers and hooking up outputs to inputs. It’s so named because it runs on Kubernetes, which we’ve been working to build some expertise in – so it seemed perfect, especially since I was looking for a nice way to run multiple training jobs and serve the resulting model!

However, the past few weeks have been a major headache, not least because my knowledge of the tech it uses was very low – but also because Kubeflow is still very, very new. Some fun issues encountered:

• Deploying using GCP’s OAuth – simply doesn’t work out-of-the-box with the current version. I’m using username/password currently
• The auto-provisioned node pools which spin up when pipelines request resource don’t have the rights to access the container registry by default: need to set the service account and some claim requirements via gcloud CLI
• Fairing, a great concept and feature which will allow Python code (notebooks, files or just functions) to be wrapped up and deployed/served in Kubernetes –
• Everything I’ve encountered has seemed very finnicky, tricky to understand and a balancing act to find the APIs which can work in each use case

This is not a criticism of the team or project – it’s being developed “in the open” by a team at Google, and I have been able to use many features to my advantage. But it’s cutting edge tech and there is much pain to endure getting going right now.

LED Saga continues

I ordered some extra-wide-angle, diffuse-lens, ultra-turbo-bright LEDs from Lighthouse LEDs, which arrived last week.

They draw less current, come with a datasheet, are nicely made and are smaller/lighter.  They do have a wider angle, and less of a ‘point’ shining out the front. However, I’m still not convinved it’s the right effect for the Flashers (old LEDs, then new ones):

And so, my thought now is to not embed the LEDs in the walls of the cube, but build a means of having them central in the interior, equally angled apart – which would mean each LED sitting at the vertex of an Icosahedron.

So next up – trying to find a way to 3D print a support structure which can hold 12 LEDs, plus individual resistors, such that they are positioned like that.

Oh, and getting the model hosted and served correctly using Kubeflow / Fairing…