Tumgik
simplythetest · 1 month
Text
The Possible and Impossible: Mathematical Thinking for Planning
Sometimes, even knowing what is possible can be big step in the right direction.
In the past I've written about impossibility theorems in mathematics. These are class of theorems that definitively conclude that, given a set of constraints and conditions, something is impossible. A straightforward example is the impossibility of tiling the plane with regular pentagons. Try finding some regular pentagon shapes and making them tile a floor; you will find you cannot do this.
On the flip side to this, there is another class of theorems in mathematics called existence theorems. These are theorems that show that, definitively, something exists given some assumptions. Existence theorems may not provide any information on how to figure out how to find that thing that exists, or even provide a clue to what a value is. They may not even guarantee there's only one of the thing that exists. The only information an existence theorem (in general) provides is that a thing exists. One example is the contraction mapping theorem; basically if a function is a contraction - ie loosely speaking, the function's input maps to a subset of the input, "contracting" the input to a smaller set - then iteratively applying the function will produce at least one "fixed point" or a single point the iteration converges to.
So what does existence and impossibility theorems have to do with, well, anything other than higher mathematics?
Human beings tend to like certainty. We like knowing certain things will happen or won't happen, and avoid situations with uncertainty. Impossibility theorems and existence theorems are ways to figure out that, yes, something is certain. Whether that thing is certainly impossible or certainly possible may help us reckon with what to do next. This certainty may provide clues to how to find a solution, or to stop looking for one, or even to reformulate our assumptions.
In any case, knowing what is possible can be a big help, at least on a psychological level.
1 note · View note
simplythetest · 4 months
Text
So You Want To Hire an SEIT
As software companies grow so do their engineering teams. A role that comes along with growing teams is the software engineer in test (SEIT), a software developer who primarily works with test automation and related testing infrastructure. SEITs can be extremely valuable to engineering teams as they can enable good automation practices for shipping quality code faster, and they can be a great touchpoint for test engineers and operations engineers to work with development teams. They can also be a bit tricky to hire since SEITs don't have quite the same skills as application developers or test engineers.
Let's take a look at how your might approach hiring SEITs at different levels.
Junior/Intern
I've worked with several interns (in Ontario these people are called co-op students) and working with test automation is a great introduction to being a software engineer of any kind.
When to hire
Juniors are great on teams with established SEITs and test engineers. In some organizations there are entire teams dedicated to test automation efforts. If you need someone to bang out many test code or take partial ownership of a large test automation project, a junior hire could be a great choice.
Like in other software develoment roles, hiring a junior usually comes with an expectation that the junior candidate will receive some menitorship and early career training and experience.
What to look for
Candidates for a junior SEIT should show characteristics of a junior test engineer and/or application developer. On the testing side, an ideal candidate should show curiosity about software or systems, the ability to develop at least basic mental models of how a piece of software might work, and good communication skills, written or oral. On the development side, an ideal candidate should show at least a basic aptitude for programming from school or otherwise. They should be able to read, write and understand code in some programming language even a language that your organization doesn't use. Previous experience working in software development is nice but not a requirement. It can be safe to assume that junior candidates have worked with at most one test automation tool and likely none.
Key questions to ask
"How would you test a pen?": a classic test engineering question to access a candidate's reasoning skills, communication skills and question asking ability.
"Write a function to reverse a list in a given language": a basic but straightforward assessment of programming skills. The language can be set by the interviewer or chosen by the candidate.
"What is something you achieved that you are proud of?": a question to get some insight into the candidate's motivations and interests.
Intermediate
Teams getting started with test automation or parts of test automation could benefit from hiring experienced SEITs. Experienced SEITs can write test code but also set up needed infrastructure and coordinate with developers, operation engineers, and testers. Intermediate SEITs may also be able to start test automation efforts for new projects as needed.
When to hire
The main reason to look for an intermediate SEIT is when automation efforts exist but have grown past being managed part time by single individuals. For example, a team may have written some Selenium-based tests to test some scenarios, and at first a small number of tests can be handled by the team or specific members of the team without a test automation speciality. Once the number of tests grow to a certain size and/or complexity, that team may want to have a dedicated person looking after them. This could also be the case with automated performance tests, service-level tests, and so on. Or a team may seek to level up on test automation skills.
What to look for
Test automation is a speciality of software development, and as I've said before (if slightly reworded) SEITs are developers. Evaluating them as you would other intermediate developers isn't unreasonable. Look for one or more roles where they worked as SEITs, DevOps engineers, or other forms of automation. Note however that SEITs may have written many, many lines of code but aren't traditionally skilled in areas of programming. Most SEITs wouldn't be able to code low level algorithms like linked lists or binary trees and may have never really used map/reduce/fold approaches. Intermediate SEITs should be able to discuss tools they've used well, and strong intermediates can identify differences between tools and levels of testing. And they can definitely write tests; any intermediate SEIT should be able to write a few tests for at least one layer of an application, such as unit or component tests, API level tests, and/or end-to-end tests.
Key questions to ask
"Write three automated tests in a given language to test this class": This question assess a candidate's knowledge of test development code writng
"Here is an example login test. How would you improve it?": A possibly fun exercise that helps evaluate a candidate's knowledge and technical communication skills.
"Suppose a software bug is found using test code that you wrote. What would be your first reaction?": Teams handle bug reporting in various ways with varying levels of success, so this question can shed some light on a candidate's experiences.
"Describe a time that you and a team mate - could be an app developer, tester or SEIT - disagreed. How did you resolve your differences?": Classic interview question for verifying what a candidate's personality is like.
"What's your favourite continuous integration tool and why"
Senior
If you work for a organization that has experienced growth and is now "established" in some sense, you may find some senior SEITs or other roles similar to that. In medium- to large-sized companies - companies a typical person may have heard of - there are often whole testing teams and sometimes engineering teams dedicated strictly to internal infrastructure. If you consider automated tests as infrastructure, this means you'll need dedicated folks to oversee project health. This includes writing and maintaining test code, but also includes managing people and computational resources. Good SEITs know how to put things together from a testing standpoint, and how to organize accordingly.
When to hire
This is definitely a bit of a "I'll know it when I see it" situation. Teams can often get pretty far with hiring and retaining good intermediate SEITs or a combination of SEITs, testers and DevOps folks. In many larger organizations, test frameworks can grow to require their own dedicated team leads and managers. This is often where senior SEITs come in. Another scenario is where test automation efforts expect to ramp up, and having someone knowledagble in test automation can guide the process.
What to look for
Senior SEITs have worked with multiple forms of test automation at most or all layers of an application. They will likely be well versed in more than one area of test automation, and have built test frameworks from stratch and worked with existing test frameworks. Depending on the circumstance, they also may have some team lead or engineering management experience, and so may be able to work with more junior members on a team in addition to coding and configuring infrastructure. One of the most valuable aspects that senior SEITs can bring to the table is an informed opinion on test automation topics.
Key questions to ask
"Suppose you were asked to review a test suite of 1000 tests. How would you approach this?": This question gets at how a candidate approaches a relatively complex situation. The idea here is spur discussion more than provide a flat answer.
"What's your preferred test framework and why? What's your least preferred?": In light of hiring someone who brings experience and opinions, this question asks for some of those opinions directly. It can also be a good indicator of communication abilities and temperament.
"Describe a time that you and a team mate - could be an app developer, tester or SEIT - disagreed. How did you resolve your differences?": Again, a classic interview question that could be critical if the candidate in question has to manage people.
"Are you willing to mentor junior app developers and SEITs?"
"Explain the value of end-to-end testing to me if I were the CEO of the company": Test automation can seem like an esoteric speciality to those not in engineering organizations. Senior developers of all kinds (in my opinion) should be able to talk in business terms in addition to technical terms.
Happy testing!
0 notes
simplythetest · 7 months
Text
Breaking Down Problems Like a Mathematician
This week I changed a toilet seat on my main floor toilet. This was a substantial achievement, and I was able to do this thanks to my extensive mathematical training.
If you've ever changed a toilet seat on a standard toilet (in North America, anyway), you'll know it is fairly straightforward: undo the screws or latches on the old seat, remove the seat from the porcelain, remove any screws or nuts, and attach the new seat. I've done this a few times in my life and it is not too difficult.
However, this toilet seat was not easy to replace.
It was a style installed by the previous owners of my house that did not have easy to remove latches or nuts. The hinges were chrome and only attached by thick screws that went through the porcelain and had plastic nuts that held the screws and seat in place. The nuts were fastened extra tight, and could would not budge. It took me multiple attempts with multiple wrenches, and I could still not remove even one nut.
The turning point was realizing the entire problem (remove the old seat and install the new one) depended on solving one small aspect of the problem (remove one nut from a bolt). If I could solve this one small problem, I could solve the entire larger problem.
In mathematics sometimes a large problem can be reduced to a smaller problem or to a set of smaller problems. For example, properties of square matrices are often related, and knowing one aspect of a square matrix - for example, that it has a determinant of 4 - means you know lots about the matrix in question. With a determinant of 4 (aka not zero), we know the matrix has full rank and that it is invertible. Many problems in dynamical systems theory come down to knowing the eigenvalues of a square matrix, so even knowing what the determinant is can help provide information about the entire system.
In my case, I knew I had to remove the plastic nut. Using a hammer and chisel I hacked off the nut, which loosened the bolt enough that I could shift and remove the second nut on the other side, and in turn, remove the old seat. I was able to reduce my larger problem into a smaller one.
This shift in thinking also had a psychological benefit. All of a sudden a difficult multi-step problem became a focused single-step one. I moved the problem forward.
Math is surprisingly helpful to learn as a way to learn problem solving and thinking, even in situations that look like you're going down the drain.
Friendly Announcement: If you liked this post or ones like it, sign up for my math newsletter
1 note · View note
simplythetest · 8 months
Text
Thoughts on Targeting Quality 2023
It's been a heck of a few years for me, personally and professionally. One thing that I am grateful for is returning to Targeting Quality 2023, the annual conference held by KWSQA. This year the conference was held in Cambridge, Ontario. Here are my thoughts on it.
Overall
The testing community is still a great community. Attendees were engaged, and even as someone who's done primarily test development for his career I was still able to have great discussions with testers about all kinds of topics.
The Talks
The first talk of the day on Presentation day that I saw was Bas Dijkstra's How well do you wield your tools? - The importance of craftsmanship in test automation which was a part of the automation track. One of the key messages in this talk was that test automation can and should be treated like production software development, which is something I've been saying for a long time. It is good to hear that this message is still coming through strong.
Test automation is definitely a mainstream topic in software development, but there's still so much that can be done by application developers in terms of test automation. Concepts like property-based testing, contract testing and security testing are all highly relevant for test development.
Fuzz testing: still not a common topic outside of the security engineering world
Another talk that I enjoyed was The Automation Firehose by Thomas Haver of M&T Bank. This talk gave a great overview of how to think about growing and scaling test automation teams with a superhero theme.
Ben Oconis' talk on breaking down silos in organizations sparked some thoughts from me and folks online about the role of "test specialist" in software organizations. Ben's talk was really good at outlining some organizational aspects to quality, which aren't necessarily "testing" skills but important nonetheless.
On this note, software testing is still a great place to learn about good team building practices. Lean Coffee, for example, is an excellent format for getting engaged folks to talk about a few different subjects in an effective (and time efficient!) manner. There were two sessions of lean coffee lead by Lisa Crispin and Janet Gregory. I attended one of these sessions, and I can't recommend trying lean coffee enough.
My favourite talk of the conference was the keynote, simply entitled STAIRS! by Jenny Bramble and Jenna Charlton. Entertaining and insightful, Jenny and Jenna show great examples of cognitive biases and how they can effect how products get designed using examples of stair designs (some of which were hilarious bad). This keynote showed that people in the testing community are digging into concepts such as design thinking and how that impacts testing work and quality, and doing so ahead of other in the software development world.
The Conference
I have to say: this is one of the best run conferences I have attended. Targeting Quality was 2 days this year, one day of workshops (which I did not attend but heard good things about) and one day of presentations.
The venue was well-prepared and suitable for the purpose, right down to stacks of pens and notepads available all over the place.
Food and drink was great, including delicious Portugese-style custard tarts at breakfast (a local delicacy, in my opinion)
The talks and schedule were well-organized and everything ran pretty much on time throughout the day. This is an underappreciated aspect of any conference, large or small.
Overall, this conference offered a lot of benefits for attendees and speakers. The hall-way track was lively, talks were great quality and speakers were well taken care of. The KWSQA board and Tina Fletcher should receive a round of applause for a great job.
1 note · View note
simplythetest · 8 months
Text
The Task Is Not The Tool
Really, the title of the post says everything. Using a specific tool is not the task to complete. It is the tool.
This comes up often in software development, where folks like to discuss and plan around a tool instead of a task, but an analogy I've been thinking about is dishwashers.
I have a dishwasher and, by far, it's my favourite kitchen appliance. Dirty dishes go in, I turn it on, and a little while later, clean dishes come out. It works, it works well, and it does what I need it to do. One day, I was thinking about the role of being a dishwasher in a restaurant, which is a traditional "entry-level" role in the hospitality industry. I wondered: Why don't restaurants simply use one or more great dishwasher appliances instead of a full-time employee dishwasher?
This thought (which wasn't too well-considered) lasted a few minutes before I realized the answer: there's more to keeping dishes clean than washing them. Dishwasher employees need to clean guest flatware and tableware for sure, but also pots and pans and such for the back-of-the-house kitchen as well. These items need to be cleaned, and allocated back to where they should be, and this should be done in a timely way. Cleaning a dish is only one aspect of this job. The task is to keep items in a kitchen clean and ready as needed, where an automatic dishwasher appliance is a tool to help accomplish this task.
The same goes for test automation approaches, for example. Choosing Selenium over Playwright for browser testing is making a decision about a tool, but deciding what kinds of tests or aspects of a software development process should be automated is (likely) the task.
Keep this in mind when thinking about tools and tasks.
1 note · View note
simplythetest · 9 months
Text
The Hottest Club in Town: Cybersecurity
If you're looking for a good time, check out the hot new tech subject, Cybersecurity.
(I'm channeling my best Stefon here.)
Seriously, this topic has everything. Two talks in this area from this year definitely have me thinking that when it comes to cybersecurity, the truth can be stranger than fiction.
Earlier this year, I attended a talk at BSides Rochester entitled "TikTok Under Attack: Attacker Uses a Popular TikTok Challenge to Lure Users Into Installing Malicious Package". I was intrigued because I do enjoy TikTok, but was not prepared for the roller coaster ride of this talk.
The overall story goes like this: the speaker is a security researcher who was interested in starjacking, a potentially malicious process where a given GitHub repository is shown to have a much higher number of stars/stargazers than it actually does. This can be exploited by poor controls on package managers such as NPM or PyPI which don't necessarily validate a GitHub repo's metrics. Here's a good intro to the topic.
This researcher was finding ways to understand how starjacking works, and found that there were malicious PyPI packages that if installed, will enable starjacking on particular repos. Installing a PyPI module does require some Python knowledge - it's not quite as easy as clicking a link on a webpage - so the attackers had to find a way to get individuals to install this mysterious PyPI package.
This is where TikTok comes in.
The researcher found a challenge on TikTok called the Invisible Filter challenge. The idea was that TikTok created a filter that makes a person in a video "invisible" (you can only see a rough outline of a human while the shape blends into the background). The challenge was to create videos and have viewers guess who the invisible person is, what they're wearing, etc. A variation of this challenge was for the person in the video to be naked so to "appear naked" on TikTok without actually appearing, er, naked. The attackers exploited this by posting links to their malicious PyPI package saying this Python package could remove the invisible filter showing the person in the video. This caused a bunch of folks (referred to as "creeps" by the speaker) to download and install the malicious package, thus facilitating starjacking.
That talk was a wild ride. Where it started and where it ended up were two different places.
I thought for sure this was a one-off style of presentation (PyPI! Starjacking! TikTok! Nude people!).
But wait, there's more!
During this year's BlackHat conference, a PhD student gave a talk called "Houston, We Have a Problem: Analyzing the Security of Low Earth Orbit Satellites". Effectively, the researcher asked "How secure are in-orbit satellites?" and the answer is "not very".
While I didn't see this talk myself, I can imagine this having the same flow as the TikTok talk. For example, the researcher found that many security features of satellites are non-existent, likely based on the thinking that attackers on Earth couldn't access them in any way. That turns out to be false, and even cloud services such as AWS offer Ground Service to low-orbit satellites. You can even get started for free.
Again, this talk had everything: basic hacking techniques, cloud providers, satellites...in space!
When I think of cybersecurity, I typically don't think of talks that offer stranger than fiction style narrative, but it may be the best place to find some such tales.
0 notes
simplythetest · 10 months
Text
Doing the Work vs Finding the Work
Many years ago, I took two courses in graduate-level mathematics as part of a summer school program at Dalhousie University in Halifax, Nova Scotia. It was an enlightening experience that very much enjoyed, learning some math on Canada's East coast (I highly recommend visiting Halifax in the summer).
In one my courses, we were given an assignment to prove a result from probability theory. The proof was relatively difficult but not impossible, and those in the class were not necessarily experienced in intermediate probability theory (myself included). The professor mentioned that there was a proof of this result published in at least one textbook, if not other papers and books. He also said he would accept a reference to this proof from a source, instead of proving this result directly ourselves. This was the first time I ever had a mathematics professor (or teacher of any kind!) tell me that I didn't have to prove a result myself if I could produce a valid reference to a proof.
So I went to the main library on campus, did some digging, and found a proof in a textbook. I took down the reference, handed in my assignment, and received full marks for that section. Mission accomplished.
Engineers, software or otherwise, may have heard the phrase "a few hours of research can save a few weeks of development" and my experience with this proof was my first real experience with this line of thinking.
Sometimes there's no reason to re-invent the wheel if you can find the work done somewhere else.
0 notes
simplythetest · 10 months
Text
Don't Build a Pseudo-Number Generator
This is a part of a series of things developers should not build.
Often, developers need to write code that has some element of randomness. In game development actions may depend on a dice roll or coin flip. In scientific programming and simulation generating random fractional values within a range (between 0 and 1, for example) is a common task. From a design perspective, perhaps a developer needs to randomly select an image from a set of images to present in a front-end. In any case, a developer may think "I know, I'll write some code myself to generate some random values".
This is -- with some narrow exceptions -- not a good idea and a developer should not do this and instead use a pre-built pseudo-random number generator library. There are libraries for this task in most programming languages. Failing that, there are CLI utilities that can be called from the OS level.
Let's take a look at why this is the case.
The Theory is Hard
First, what's the difference between random and pseudo-random anyway? Isn't this just a technicality?
The short answer is: no, it is not just a technicality.
Randomness is a difficult topic for human beings to understand. Truly random events have zero relationship with one another, meaning that one event has zero impact or influence on the next one. If I roll a six-sided die and it turns up a four, then rolling again has no impact on the next roll the next time if the rolling event is truly random.
Now consider about a computer program or script. It needs to run from the beginning to the end. Code runs in a predictable and deterministic way. By definition running a program from beginning to end cannot have any randomness. Computers can't stop and roll a die.
Luckily, there's a workaround: create a function for generating numbers that appear random, statistically. Such a function returns the same output for a given input. However, each generated number is "random-enough" to simulate a random event such as dice roll. Mathematicians have worked on creating algorithms and functions that output random-seeming values called pseudo-random numbers that have good properties for being close enough to random. These include algorithms such as linear congruential generators and the widely implemented Mersenne Twister algorithm.
(As a side note, if true randomness is a hard requirement in an application such actual lotteries using dice rolling, there are approaches that use physical phenomena to generate true random values. See RANDOM.org for more information.)
Given the theoretical issues, the creation of pseudo-random number generator (PRNG) algorithms provide a path forward. Sadly, PRNGs do not end the problems developers face.
The Practice is Hard
Implementing a PRNG seems straightforward. Find an algorithm, pick some values for various aspects of the implementation, and execute away. Many PRNG algorithms are effectively difference equations that start with a given value and iterate through a provided number of times. Easy right?
Well, no.
The first issue that arises is computation time. Generating a single value from a PRNG is fast, but what about generating 100 values, or 1000? In practice, applications require a random value on every command or execution which could mean that many, many random values need to be generated. If each generation requires many, many iterations of a particular algorithm this could lead to non-trivial execution times. In game development, this could be a deal breaker, since performance is crucial. Often PRNGs need to be fast at runtime.
Computer hardware has gotten fast thanks to Moore's Law so even executing a large number of iterations within a function is pretty fast. However, speed isn't the only consideration for PRNGs. There's also the aspect of being "statistically random" which means choosing a "good" PRNG with good configuration.
An infamous example of choosing a "bad" PRNG algorithm was RANDU. RANDU was based on a linear congruence generator as mentioned above. While RANDU generated approximately random values, it had some obvious flaws. For example, RANDU only generated odd numbers. As well, after a while generated values would repeat, meaning there was a pattern that did not appear random at all. Worst of all, RANDU was taken an implemented in a variety of settings. Note that this was a vetted algorithm by smart folks.
While it is definitely possible to test a random-number generator for correctness and performance, PRNGs are sensitive to configuration and there are many pitfalls that a developer may encounter if they are not familiar with the problem space of numerical analysis and computational mathematics. Which in my experience is not a lot of developers.
However, smart folks have learned from their mistakes and now there are off-the-shelf PRNGs using vetted algorithms in most mainstream programming languages.
It's all good, right?
Don't Forget About Security
The last and possibly biggest issue with using PRNGs in code is security. If an attacker can determine the PRNG values being generated by a program, that attacker can exploit this knowledge. A helpful example is an online poker site. If each card that turns up is chosen using PRNG algorithm, then a player who knows what value the PRNG will output next can use that information to determine every single card dealt to every player. This knowledge can easily be exploited to manipulate each hand.
One of the reasons that RANDU was such a disaster was that it was not secure and it was widely used. Once an attacker know a piece of software is using RANDU they can exploit that information for other malicious purposes. Using a PRNG in a local toy piece of code may be completely fine but using an insecure PRNG in production could lead to major security holes.
Again this is not a scenario that many developers consider when deciding to use a bit of randomness in their code.
Mercifully, there are well-tested cryptographically secure PRNGs available in many programming languages.
In summation, choosing a PRNG built and maintained by experts is a better bet than rolling your own.
1 note · View note
simplythetest · 11 months
Text
Programmers, Don’t Build These
In the spirit of “You can’t buy integration” here are things that a programmer should never build:
A Pseudo-Random Number Generator
A Cryptographic Library
A Numerical Integration Solver
A Fuzzer
Each of the reasons why are straightforward: building each of these from scratch is loaded with pitfalls that could cause problems that far, far outweigh true benefit.
1 note · View note
simplythetest · 1 year
Text
Choosing Developer Events: The Christoper Plummer Approach
How you decide what events to go to?
This is a tricky question for most people, but even more so for people who go to events as part of their job. For example, developer relations advocates often present at conferences to tell people about their cool projects (like for example a AI-guided fuzzing tool). However, there are only so many hours in a day, and only so many events someone can attend in a year. How can you choose the events that work best for you?
One approach that I think works well is to take inspiration from one of the greatest actors of a generation: Christoper Plummer.
The late but great actor Christopher Plummer took every kind of acting role you could imagine: large scale blockbusters musicals to European dramas to Shakespearian plays (and even some lesser quality films). When asked how he chooses roles in an interview, Plummer said a role needs to have one of three characteristics: the role is interesting enough, pays well, or is in a great location.
This is how I think about events, and I think the Christoper Plummer rule works well. Stated for choosing developer events, you should consider going to an event if the event
is interesting,
pays well, or
is in a great location
Interesting Role (or Event)
Developer events and conferences come in shapes and sizes and cover all kinds of topics, from the most narrow programming construct to the broadest area of industry. Depending on your interested or priorities, there are likely a few "major" events or conferences that take place on a regular basis. For the Python world there's PyCon, for the infosec world there's DEFCon, and so on. These events draw a lot of people and event simply showing up without a plan may still be beneficial. These major conferences and events are definitely something to consider, even if you've never attended or aren't overly familiar with them.
Pays Well
Sometimes conferences will pay for speakers to attend, and in some cases this includes food, accommodation and travel expenses. Similarly, most conferences look for sponsorship from potential partners and vendors and in turn provide conferences passes. If you have the inclination to speak to people, from onstage or from a table, you may be able to attend a conference at very little out-of-pocket cost. Who doesn't love free stuff? Finding opportunities to go to conferences at little to no personal cost might help make them more attractive to you.
An Interesting Location
Have you ever wanted to visit Disney? Then going to a conference or event in Anaheim, California or Orlando, Florida might provide the perfect reason to visit. Many places draw visitors for business or pleasure based on local culture, commerce or climate. If an event happens to take place in somewhere interesting to you personally, it might tip the scales in your favour of attending. A great location also makes a less valuable event seem a bit less bad.
If all the world's a stage, then consider these three things when choosing where you'd like to be a part of the play.
2 notes · View notes
simplythetest · 1 year
Text
Conference Recap: SeleniumConf 2023
This year, I had the honour and pleasure of attending SeleniumConf 2023. This was the first in-person, full-blown Selenium conference since 2019. This year the conference was held in Chicago, Illinois which is pretty convenient for me, since I live a couple of lakes over in Toronto.
Overall, it was a wonderful conference with great people talking about a great piece of software. Here are some thoughts and highlights:
Selenium State of The Union
As this was the first Selenium Conference in a while, there were several notable changes from the Selenium project itself. One change is that now Diego Molina of Sauce Labs is the main maintainer of the project, taking over for the esteemed Simon Stewart. Diego gave the Selenium State of the Union keynote to open the conference (side note, all conference talks are now available on the Selenium channel on YouTube).
One happy note about this opening keynote is that there was a lot of focus on the Selenium community and providing encouragement to contribute to the project in some way. The Selenium community is fairly large, particularly if you include end users of the Selenium WebDriver, Grid and IDE, so it is good to see the Selenium contributors reaching out to the community for help. The next year or so definitely seems like a great time to contribute to the Selenium project if you've ever wanted to. Diego's keynote highlights this pretty well.
Marketing Releases vs Technical Releases
Another big happening in the Selenium world since the previous SeleniumConf was the highly anticipated release of Selenium 4, which including solidifying the W3C protocol specification, relative locator strategies, and some other technical updates. A lightbulb moment for me was during a conversation with longtime Selenium aficionado Adam Goucher where he mentioned that while Selenium 3 and 4 were technical releases because of these technical features, Selenium 5 could be a marketing release that has some really great UX and quality-of-life updates. This meant that while Selenium versions 3 and 4 were mainly about technical changes in the application itself, Selenium 5 could be a release mainly focused on cool stuff that end users want to see. Software is like this, whether open source or not: there are important technical aspects of a piece of software on the inside, and there's the shiny fun stuff on the outside. Both aspects are critical for high quality software but sometimes the technical takes precedence over the shiny and vice versa. Thinking about both is what makes good software great.
The talks were good, actually
Chatting with a few people in and around the conference, some people mentioned how the talks overall in both tracks were quite good, and I have to agree. While I've seen some talks that fall into the "two stars, would not watch again" category (which is another blog post in itself), the quality of talks overall at SeleniumConf this year was quite good! Some selected talks that I enjoyed:
The second keynote of day one was from Erika Chestnut who gave an excellent talk on building a culture of quality as finding missed opportunities. This is a great talk for test developers and non-test developers alike. One analogy that struck me was Erika using the analogy of a culture without quality as cooking without seasoning, which I'm still working my mind around.
Noemi Ferrera aka TheTestLynx gave a great talk on web crawlers, which provided a nice overview of how to think about web crawlers and different approaches for different contexts.
A talk that I personally enjoyed was Benjamin Bishoff's talk on identifying code smells. In efficient fashion, he presented twenty-one (!) object-oriented programming code smells and provided examples and explanations of each one during his talk. You can read more about his time at SeleniumConf.
Another talk that I enjoyed a surprising amount was Shi Ling Tai's live demo of difficult to test web UI elements using UI-licious. The live coding examples went very well considering she presented cases that are traditionally "flaky" for web automation, and the talk was crisp and fun.
The second day's keynote was given by the always great Mark Winterinham who talked about what exactly people do in test automation. This was a great keynote for test automation specialists since it gave a description of automation as a story that we tell.
and of course, I have to mention my talk, Selenium in the Clouds which went well.
Tools are tools are tools
A theme that I found interesting that arose through several talks was the discussion of tooling. A few times I heard from people who were not me phrases like "Selenium is only a tool" or "learn to use different tools for different purposes" and so on. To me this is a nice trend to see at conferences like SeleniumConf, since many folks who use Selenium use other similar but different tools as well. The idea of "only" using this tool or that, or test automation "never" using this tool or such, is one that is frankly incorrect and unhelpful. Learning to think critically of tools in your context is an important lesson and I'm glad to see this at such a mainstream software conference.
I really love Chicago
As I mentioned at the top of this article, Chicago is pretty close to Toronto which was a motivator to go to SeleniumConf this time around. However, I've been to Chicago before and this trip reminded me of how lovely a city Chicago is. It's a large city but it is also walkable: I was able to find everything I needed within a short walk of the hotel venue. The architecture of Chicago is beautiful, diverse and yet somehow coherent. Plus, there's deep dish pizza.
Between the beautiful city, the great venue and the excellent conference staff and volunteers, SeleniumConf 2023 was a hit.
1 note · View note
simplythetest · 1 year
Text
Now and Then Degrees
If you had to ask me what courses I've taken over my life have been the most helpful or impactful, I'd have to say grade 10 English and my introduction to topology course I took in my undergrad math program. Yes, I'm surprised too.
Both of these courses taught me somethinng that's applied in several times in my life in broad contexts. Grade 10 English taught me the importance of naming things (not calling your essay "Hamlet Essay" but giving it some interesting title) and when writing or trying to express a point, repeat the point often (more often than you think!) to really drive the point home. Topology taught me, well, some topology but also how to deal with a topic that you find difficult to understand or struggle with. Algebraic topology is still probably the most crazy difficult topic I've learned and I may never understand it, but now I feel more comfortable saying "I don't understand this" when I encounter difficult topics.
I often think back to my math courses that I took since they were the courses that taught me how to think and problem solve in a more abstract way. It's taken many years to get there, but I'm finally appreciating my time learning mathematics beyond the actual topics themselves. I think when people choose a post-secondary program, there can be a tendency to focus on the "now" aspect of the program. What job prospects will this program give me? Is this something I'm interested in now? How does this program fit in with my current plans? Programs that are job focused fit well into this thinking, like engineering, accounting, medicine and so on.
Some programs are more "later" programs, where it takes time to realize the benefit and see how this program fits into your life. A pure mathematics degree might fit into this way of thinking well, similar to a degree in literature or history. The value may not be immediately obvious, nor are job prospects, with a later degree. But the value may far exceed your expectations.
0 notes
simplythetest · 1 year
Text
Test Automation Patterns: The Test Automation Trifle
As a test automation professional and a foodie living in an excellent food city, it was only a matter of time before I write about some kind of test pyramind food metaphor.
We've seen the test automation pyramid, and its cousins the test automation cupcake and test automation ice cream cone. Let us consider the test automation triffle.
Triffle is a dessert common in the British Isles. It's a layered dessert that is both pudding-like and cake-like, depending on the composition. Generally it contains an egg custard, some kind of fruit (preserved or fresh) and a sponge or cake layer. I've seen several variations, I'm happy to report.
The key to a good triffle is to have enough of the best layers. If you like custard, put an extra layer of that on there. If you're more of a sponge fan or want something lighter, add extra layers of those and don't add so much sweet or heavy elements. Brandy is a popular ingredient but far from required. And so on.
This is kind of like test automation. The best developers I've known have all said some variation of "labels and test types don't matter too much, just test the thing you need to test". The notions of unit, integration, end-to-end, API, service, and so on, aren't as important as testing the essential important part. If you're a web developer, that's going to look different from an embedded systems developer, which will look different from a Python-stack towards something solely on the JVM. You want to test the critical layers, whatever they are for what you're looking at.
A bit like triffle.
The test automation pyramid is a very helpful construct, trust me. It's a good guideline to think of testing generically. But you can always adjust to get the best bits.
0 notes
simplythetest · 1 year
Text
When Examples Attack: Good Practices For Writing Example Code
Picking the right examples matter. Google recently learned that they matter quite a bit.
In the past I've written about writing code samples and some good practices (I think) to follow. These include knowing your audience and keeping your examples up to date. Having good examples of what your code - or product - can do is essential for getting some attention in a competitive marketplace.
I suppose I should've mentioned that your code examples should be accurate and have outputs that are correct but I assumed that was obvious.
This week, Google announced a new AI chatbot called Bard. This chatbot could answer all sorts of questions in the same manner as the previously announced ChatGPT. However, in Google's ads they decided to use an example of Bard answering a question about the first images taken in space. This resulted in an response that was factually incorrect. Not a good look for an AI chat assistant. As a result, Google lost billions in trading.
When I read this story about Google's unfortunate ad choice my first thought was something along the lines of "why didn't anyone test this example?". Even one person validating the ad copy and fact checking before going live could've caught this. So perhaps this illustrates an additional good practice for sample code:
Test your example code to make sure it works expected.
This kind of advice might just prevent a costly billion dollar mistake.
0 notes
simplythetest · 1 year
Text
Quality as Priority
You get what you prioritize.
Lately I've been thinking of quality in general when it comes to products. One of the most influential pieces of writing to me on quality was this article on the case for bad coffee. I've blogged about this article before in the context of software development methodologies, but I think illustrates one good way to think of quality, which is quality as priority.
The late and beloved Gerald Weinberg defined quality as "value to some person", which may be refined to talk about the value a stakeholder has, or value to some person at some time or what have you. This is an excellent starting point for thinking about quality, and many folks in software development and elsewhere have considered Mr Weinberg's wise words. The definitely of quality is a bit vague, however. Suppose you want to improve or evaluate the quality of something. Where does this definitely leave you?
In the Case for Bad coffee, the author Keith Pandolfi discusses his love affair with creating specific coffees in particular styles, taking care in the preparation and serving of the coffee. He later finds his passion for taking time to create a single origin espresso gave way for being able to have cheap mass-produced coffee readily available to drink. In both cases he finds value in varieties of coffee that are quite different. So what happened?
The difference is in his priority of what a quality coffee is.
First he finds high quality coffee in carefully prepared specialty coffees in comfortable environments. In other words, he prioritized the experience and/or sense of taste of coffee. Then, over time, he finds that having coffee that may not taste that good but is available whenever and wherever he is. In other words, at this point his priority changed to having access and availability to coffee. His priorities changed and so in turn did his view of value and quality.
This thinking goes beyond coffee or even hot beverages in general. Often, when looking at similar types of software there's some variation in the quality of these pieces of software. The software is a result of priorities made by the creators. If a software has - for example - poor user experience for Windows users, it suggests that Windows users weren't prioritized or even seen as a priority. It's not that the software is "poor quality", per se, but more that the creators had different priorities than what Windows users may have. Similarly, improving the Windows user experience could be improved by prioritizing it.
I think this is a helpful re-framing of Weinberg's notion of quality, which I will mention was only one of several excellent ideas that Weinberg put together.
0 notes
simplythetest · 1 year
Text
Fuzzing is Cool, Actually
TL;DR: Fuzz testing is really cool.
I've been interested in test automation for a long time, and one area in particular that I've been interested to learn about is fuzz testing. I'm now able to finally dig into this area of software engineering.
Fuzzing has been around for a while now, starting on a dark and stormy night in Wisconsin. Fuzzing is a technique for sending semi-random and possibly semi-invalid inputs to a piece of software (a function, a program, etc) to find bugs with this software. Over time, fuzzing - also known as fuzz testing - has been used to find security vulnerabilities in software in addition to purely functional software bugs. Fuzzing has also evolved from generating totally random data to becoming more intelligent about data generation and inputs. Fuzzing falls into the category of easy problems with difficult solutions that I enjoy.
One thing I find fascinating about fuzzing is that it is a classic case of test automation: after deciding what to fuzz, a developer can use a fuzzer to create test inputs and then execute tests using those inputs automatically. Effectively, automation is being used to do lots of work that a human would find tedious or even impossible to complete. Absolutely classic in the sense of using automation in testing and software quality assurance.
Another thing I find fascinating is how effective fuzz testing can be. Fuzzing is an exploratory technique since the developer using a fuzzer can control how many iterations of inputs are being sent to the software under test. In practice, this means a single fuzzer execution can take minutes, hours or even days depending on how many iterations the developer wishes to execute. In practice, however, many bugs can be detected within a minute. Such bugs may include overflow errors (heap, stack and so on), out-of-memory errors and null pointer exceptions. All of these kinds of bugs are simple to fix but can be surprisingly difficult to find manually even when using debuggers.
Lastly, and I cannot stress this enough: fuzzing is a well-established technique and subfield of computer science in its own right. The earliest fuzzers were literal homework assignments, but over the past few decades fuzzing has become quite sophisticated. There are techniques that can "guide" fuzzing inputs to find more "interesting" results (see feedback-based fuzzing), approaches to leverage multiple fuzzers at once (see ensemble fuzzing) and efforts to fuzz open source software effectively at scale. As well, specialized fuzzers have been created for fuzzing code in particular languages (C and C++ are the most common), fuzzing browsers, fuzzing web APIs are more.
Classic, effective, well-established: all things that sound good to me.
0 notes
simplythetest · 1 year
Text
Looking Back at 2022
If I had to rate every year of my life from best to worst, 2022 would be somewhere in the lower quartiles. This year was not easy, but not terrible. Here's my post on reflecting on what was 2022
General
It may seem trite, but "plan it, and execute it" is still awfully good life advice. I revisisted this advice several times this year with good results each time.
Similarly, "be proactive, not reactive" is much better advice when you're in a position of power more than weakness.
As a Canadian who has had a lot of interactions with US Americans and Europeans, let me say that US culture is exhausting.
There's a lot more to a kitchen renovation than picking out a countertop colour.
Weirdly, keeping a somewhat consistent blog for many years is actually a bit of a blessing these days.
To paraphrase Kurt Vonnegut, peculiar travel destinations are dancing lessions from God.
One of the best books I've readin a while is Braiding Sweetgrass by Robin Wall Kimmerer. It's definitely one of those "read it slow and let the thoughts sink in" kind of reads. Highly recommended.
Parenting
These two articles on the difficulty of parenting during COVID and other crises really hit home for me. One of the things that made this year difficult was simply managing parenting through illness, rotating school closures and ambiguous child care situations. This management was draining, slowly but surely, in ways that Amil Niazi and Lydia Kiesling capture perfectly.
Two year olds are a lot of fun, a lot of work, and just a lot overall.
Something wonderful about having a kid grow up is seeing how they get into the same kinds of things that you were into at their age. Video games and science are just as fun the second time around.
The notion of "bad parenting" is possibly an badly defined concept. I'm not exactly sure how to quantify how bad a parent is doing given the circumstances of the past few years.
Software Development
More and more I'm convinced that the next "big thing" in software development tooling (if it isn't the big thing already) is User/Developer Experience. There are many, many great software tools that are simply crappy to use.
GUI automation still remains one of the most difficult parts of software engineering, talking about mobile environments or otherwise.
Test automation is such a wonderful software speciality to me because it lets me play in a bunch of different discpline's backyards. For example, I would have never learned about automated cow brushes if it weren't for my test automation career.
Probably my favourite podcast as of late is Python Bytes hosted by Michael Kennedy and Brian Okken. What I truly love about it is the format: two prominent people in the Python world just talking about cool Python modules and tools. I feel like we need more of this in the software world.
This upcoming year might finally be when I dig into some genuinely new programming languages (for me). I'm cautiously optimistic.
I've thought about it, and articles and presentations with clickbait titles such as "Tool X vs Tool Y: Who will Win?" or somesuch are fine, actually.
When I'm old and grey and look back at my life and legacy, I probably will not be thinking of C++ software development.
Tech
To be blunt: there's a lot of shitty white men leaders in Silicon Valley. And it shows.
Related to the previous point: true diversity and inclusion on teams is a major, massive advantage.
One of the most interesting lessons I've learned over the past several years is that being a CEO is still a job. CEOs can be more junior or senior, have growth trajectories and learning goals, and still have day-to-day work to do.
I was a part of the great tech layoffs of '22 (my last day was during Burning Man, no less). It was a humbling experience, but it also taught me a lot about the nature of the tech industry. In a strange way, I'm grateful for this.
Once again, with some feeling: the decision to replace humans with automation is a managerial one, not a technological one.
Never underestimate the value of a good community.
Related to the previous point: just because something is difficult to quantify doesn't mean it has no value. And just because high up decision makers don't understand the value doesn't mean there isn't any.
Developers really, really hate being marketed to. Honestly, I think this is a good thing.
Something that may be missing from open source software and perhaps corporate software is the idea of helping. Having an expert or someone with experience in an area different from yours genuinely offer help can be very rewarding, but this isn't promoted too much. It's a bit of a shame.
Canadiana
This year was (in my opinion) a record low-point for Canadian politicians. They basically all decided to phone it in.
Queen Elizabeth passing finally provided two things that united most Canadians together: an elaborate royal funeral with all the commentary and commotion, and a distaste for the now King Charles III.
One of the podcasts I've listened to in the past year was Canadaland COMMONS on the Hudson's Bay Company. It was a really good introduction to a long and storied company that really influenced the development of "Canada" as a nation.
Canada really is, historically and culturally, about seven corporations dressed in a trenchcoat.
0 notes