We used 10,000 GPU-hours.

It looks like a useful answer. There’s a number and a unit. Very official.

But what does a GPU-hour actually measure? On its own, pretty much nothing. I can’t compare that number with another project until I know what counted as a GPU, whether it was a whole device or a slice, and how the authors calculated the total.

One GPU-hour is not like another

An hour on an NVIDIA A100, H100, or B200 counts as one GPU-hour under the usual calculation. That doesn’t make those hours equivalent. These are different generations of hardware with different memory capacities, bandwidth, and compute capabilities. They have tons of different features too.

The workload matters too. A newer GPU might make a huge difference for one model and barely move another. number of GPUs × elapsed time throws all of that information away. It’s pretty much the worst collapsing across dimensions you could do.

In the past, research papers have long reported CPU models and architectures because an hour on one processor doesn’t automatically compare with an hour on another. Plenty of papers still include that detail, and they should. GPU reporting deserves at least as much care.

Device slicing makes it worse

Multi-Instance GPU, usually called MIG, is device slicing. An administrator can divide one physical NVIDIA GPU into isolated instances, each with a portion of the compute and memory resources. NVIDIA’s MIG documentation says supported GPUs can be divided into as many as seven instances.

This is a good thing. Several jobs can share an expensive GPU without fighting over the same resources. Schedulers can present each slice to a job much like a physical GPU. Slurm can schedule MIG instances as individual GPU resources, for example.

Here’s where the reporting gets weird. Say two jobs each run for one hour. One gets an entire H100. The other gets one MIG slice of an H100. If the accounting system calls each allocation one GPU, both jobs used one GPU-hour. Same number. Very different allocation.

Calling these “MIG-hours” or “slice-hours” doesn’t fix much by itself. MIG profiles assign different amounts of compute and memory. Ten hours on a small slice and ten hours on a large slice still become ten slice-hours when the report leaves out the profile.

Show your math

Different people need different numbers. A cluster operator may care about how long allocations were occupied. A funding report may care about use of the physical hardware. A researcher needs enough detail to understand or reproduce an experiment. Those are different questions. They don’t have to produce the same total.

One report might count every scheduled MIG instance as a GPU. Another might convert slices into fractions of a physical device. It could count allocated time even when the job wasn’t keeping the device busy. All of those approaches can make sense for a particular job. Calling every result “GPU-hours” is the problem. Once the calculation disappears, the reader can’t recover what happened.

A universal normalized GPU-hour sounds tempting, but I don’t think it fixes this. Converting every device and slice into one equivalent unit requires assumptions about relative performance. Those assumptions change with the workload, software, numerical precision, and bottleneck. It’s problematic either way.

At least report this much

If you publish GPU usage, include:

• the GPU manufacturer and model
• whether jobs received whole devices or slices
• the MIG profile or equivalent partition size when slicing was used
• the number of allocations
• the elapsed duration
• the exact calculation behind any aggregate total

A methods section or figure caption could say:

We used eight allocations of an NVIDIA H100 3g.40gb MIG profile for five hours each, for 40 H100 3g.40gb slice-hours. We calculated slice-hours as the number of allocated instances multiplied by allocation duration.

Yes, that’s longer than “40 GPU-hours.” You can also tell what it means. The reader knows these were slices, which hardware they came from, and how the total was produced.

If the work used several device models or MIG profiles, report them separately. Don’t add unlike allocations together and slap a precise-looking label on the result.

A caption is allowed to do some work

A single GPU-hour total fits nicely on a dashboard. The useful version takes a caption, a definition, or a few lines in a methods section. That’s fine. This is what those things are for.

GPU-hours can still work as internal shorthand for one stable pool of identical hardware. Everyone in that context knows what the unit means. The shorthand falls apart as soon as the number leaves that context.

Don’t make readers guess. Name the hardware, describe the allocation, and show your math.

Related posts

• Your KPIs Aren’t Telling You Enough

The response also confirmed why I wanted to give the talk. People are constantly asking me agents and what they should be doing with them. They’re watching other researchers and developers move very quickly with these tools, and there’s a real fear of missing out. They want to try them, but they don’t necessarily know where to start or how much of the output they should trust.

My answer is to start with code you already trust, then put some AI slop around it.

The rest of this post are my thoughts about the various points I bring up during the talk.

The title was a bit of a lie

The title says you should write more code. This is technically true, but it depends on what we count as code.

Most researchers don’t need an agent generating more of the main scientific loop. That is the meat and potatoes of the project. It includes the cleaning rules, statistics, simulations, models, and transformations that can change the result. The researcher needs to understand and own those decisions.

A lot of useful code lives outside that loop:

• tests and synthetic examples
• input validators
• diagnostic plots and reports
• logging and debugging tools
• profiling scripts
• build and release automation
• documentation

People tend to skip this stuff. It takes time, some of it is boring, and it may require programming knowledge that isn’t directly related to the research question. It is also exactly the work that can make the main analysis easier to trust.

The title I really wanted was “Use AI agents to write slop to verify your code.” It probably wouldn’t have attracted 300 registrations.

I asked Claude to do my stats

I built a synthetic reaction-time study for the talk. It had 48 participant files and a 2x2 repeated-measures design. Each participant should have completed four conditions with 80 trials in each condition.

Of course, the data wasn’t actually that clean. One file duplicated an existing participant under a new acquisition ID. Two participants didn’t finish. Some had near-chance accuracy. Others had reaction times that were implausibly fast or slow.

All of these are normal research-data problems. They’re also serious enough that letting those files contribute to the final result would be wrong.

I gave the data to Claude three times.

For the first attempt, I used this prompt:

These CSV files are the results from my experiment. Do my stats for me and make a plot I can use in my paper.

This is a terrible prompt on purpose. Claude still produced a polished plot and a significant interaction with a p-value of 0.040. It looked like a finished analysis.

It turns out it had invented an accuracy exclusion, picked its own reaction-time summary, allowed comically large response times, and included incorrect trials. There were decisions all over the place that the researcher didn’t make and might not even notice.

The result was blatantly wrong, but it looked nice.

Never do this.

A better prompt still wasn’t enough

For the second attempt, I explained the study design. I described the factors, outcome, repeated-measures analysis, hypothesis, and how incorrect trials should be handled.

The code improved. Claude gave a more conservative p-value of 0.087. It also showed hints that it could produce useful checks around the analysis.

This is probably where a lot of people stop. The response knows the terminology, the figure looks good, and the result feels more defensible because the prompt was detailed.

There were still problems in the data that anyone familiar with the study would want to investigate. More context made the output better, but good prompting did not replace experience or domain knowledge.

This is also why I’m not especially interested in advice that boils down to “write a huge prompt.” Context is useful. It does not prove that the code handled the real files correctly.

Own the logic and delegate the annoying parts

For the third attempt, I supplied the core statistical code myself. It came from previous work, and I understood what it was doing. Claude wasn’t asked to choose the analysis.

Instead, I asked it to write the surrounding checks. Report any of the missing four conditions. Report files with too few trials, high error rates, or reaction times outside the plausible range. Check participant identities. Show me potential failures.

This found all eight problematic subjects, including the duplicate participant. After reviewing and handling those cases, the analysis used 40 participants and returned the expected interaction with a p-value of 0.037.

The third attempt was better because I owned more of the important code, not because Claude suddenly became more trustworthy. The agent took criteria I supplied and turned them into reports and checks. That gave me things I could inspect before accepting the result.

This is the kind of AI-generated slop I want more of.

If a validator is a bit ugly but catches a duplicate participant, great. If a throwaway plot makes a broken distribution obvious, it did its job. If an alternate implementation disagrees with my main calculation, I now have something worth investigating.

None of that code needs to become permanent infrastructure. Its job is to answer a question or provide evidence.

This works outside toy statistics

The same idea fits notebooks, shared pipelines, and SLURM jobs.

For a notebook, an agent can add data-shape checks, diagnostic cells, caching, or a small script that tests code pulled out of the notebook. For a shared pipeline, it can validate inputs, produce reports, or turn a previously discovered failure into a regression test. For a SLURM workflow, it can add preflight checks, logging, resource summaries, checkpoints, and checks that distinguish a complete output from a partial one.

The stakes change as the work gets larger. A bad notebook cell might waste an afternoon. A bad cluster job might wait in the queue for days, consume a large allocation, and leave partial files that look usable. Catching the same mistake before submission is much cheaper.

There are also obvious limits. Don’t upload data unless you’re allowed to. Run unfamiliar generated code in a sandbox, container, or virtual machine. Keep a human involved when mistakes can affect results, people, infrastructure, or budgets. You own the problems even when an agent wrote the code.

Try one check

Don’t start by giving an agent a new analysis and hoping it gets everything right.

Take a script, notebook, or pipeline that you’ve already used enough to understand. Something battle-tested. Ask the agent to add one check that doesn’t touch the meat and potatoes.

You could ask for a synthetic case where you know the answer. Ask it to verify that every participant has the expected conditions. Have it make an exclusion report, plot a suspicious distribution, or write a second version of a calculation so you can compare the outputs.

Review what it writes. Run it. See whether it tells you anything useful about code you already know.

That is a much safer way to learn what agents are good at, and it produces something useful even if the generated code is disposable. The point isn’t to trust AI more. The point is to cheaply make more of the stuff that helps you trust the important work.

This webinar compared ways to group lots of serial tasks into manageable jobs. I covered when ordinary submissions stop making sense, how array indexes can drive repeated runs, and when it is time to use a serial farm.

Abstract

Small jobs are convenient to submit individually, but at scale they can overload a scheduler, inflate queue times, and ultimately reduce throughput. This webinar examines practical strategies for consolidating large numbers of short tasks, including job arrays, task-bundling techniques, and wrapper-based aggregation. We will discuss how these approaches differ in overhead, portability, scheduler behavior, and job-failure handling. The session will also provide guidance on recognizing when packaging options no longer yield sufficient throughput and when transitioning to a serial-farming model becomes advantageous. Serial farms can mitigate scheduler pressure, improve wait times on busy clusters, and offer more predictable performance. Examples will be provided throughout the webinar and shared on GitHub for future reference.

The SHARCNET seminar archive has the event listing.

I covered job history, account priority, wait times, and resource-use reports. The examples looked at CPU, memory, and GPU requests that did not match the work being done. The point was to check the evidence before changing a job request.

Abstract

Researchers often leave resources on the table when specifying their job requirements on the national systems. This talk builds on previous sessions and uses the Digital Research Alliance of Canada’s User Facing Portals to explore what different types of jobs look like when they waste resources. Demonstrations will include interactive jobs, parallel jobs, GPU workflows, and more. With more accurate job specifications, researchers can expect shorter wait times and more throughput on any general-purpose system.

Download the slides.

This Compute Ontario Colloquium was an introduction to Scalene. Scalene separates time spent in Python from time spent in native libraries, and it can also profile memory and GPU use. I ran it on the Alliance systems from a Jupyter notebook, then compared native Python, vectorized code, Cython, and just-in-time compilation.

Abstract

Python is a language developers choose to write in for convenience rather than speed. However, speed can be recovered by offloading calculations to libraries which leverage lower-level languages like NumPy, Cython, and more. Scalene is a high-performance CPU, GPU, and memory profiler which can illustrate where code should be passing calculations to other libraries for significant increases in speed. Scalene also includes support for Jupyter Notebooks, OpenAI suggestions for vectorizing code, as well as a significantly lower overhead and higher accuracy than other profilers. This talk will introduce the concepts required for understanding why external libraries are faster than native Python, interactions with approaches such as Cython and just-in-time compilers, as well as a live demonstration of Scalene on the Alliance systems inside of a Jupyter Notebook. Familiarity with Python, virtual environments, and Jupyter notebooks will be assumed.

The slides, notebooks, and examples are available on GitHub.

The second part was figuring out how to deploy the package for others to contribute to, as well as install on their own specific HPC clusters. Some quick searching revealed that PyPI and pip were the way to go.

To make a long story short, here are a few references that made it approachable:

• Editable/interactive mode for pip.
• Cookiecutter.
• Virtual environments.

The package is still in use today inside SHARCNET and has also received development support from WestGrid, Calcul Québec, and MILA.

ViewClust can be found on GitHub. Its cousin package, ViewClust-Vis, implements several summary figures.

As such, this talk was created to help direct potentially abstract development efforts toward optimizing for the feature sets that are most available on an HPC cluster.

Below is my abstract for the talk as well as the recording:

“Determining the optimal job configuration for a given workload on HPC systems can be a difficult problem. Researchers often have different job needs, different responsiveness requirements, and different scales. This webinar will discuss these differences and how to investigate making the scheduler as responsive as possible. Topics will include whole node scheduling, by core MPI jobs, GLOST, META, and more. This presentation will assume basic knowledge of job submission, and the Linux environment. Practical examples will be discussed and used as introductions to new tools to maximize performance on the general purpose systems. Open questions will be allowed at the end of the seminar.”

During a fun Friday chat, a colleague remarked that Python was, on average, 40x slower than C++. I defended my current language of choice, saying it was better than that, surely. To make a long story short, I was wrong. It really is about 40x slower depending on the problem. Determined to prove myself capable, and my language of choice a bit more defensible, I decided to look into ways to make Python faster.

I eventually landed on Cython. It turns out the best way to make Python faster was to use as much C++ as possible.

Below is my abstract for the talk as well as the recording:

“Often we write programs in Python for convenience, not for speed. When work becomes elevated to High Performance Computing (HPC) environments, speed once again becomes a concern. Cython is an extension of Python which allows functions to be compiled as C (or C++) and recover the significant performance trade-offs of Python. Cython achieves this by supporting calling C functions, declaring of type information, as well as providing access to C++ STL functionality. Popular packages and libraries that take advantage of Cython include: TensorFlow, OpenCV, NumPy, Pandas, and more. This webinar will cover a basic introduction to Cython, a demo translating vanilla Python into Cython, followed by a short demo of how to run Cython in our own Compute Canada HPC environments. Experience with Python will be expected, while familiarity with C/C++ and Jupyter notebooks will be helpful. Webinar material and code will be made available on GitHub for reference.”

Related posts

• Revisiting Cython: Is It Still Effective?