Meet Your Instructors Series – Tim

Hello! This is Rachel and I will be hosting a series of Q+As with your valued instructors so that you can get a glimpse into their specific career backgrounds, teaching styles and processes.

We will be starting with our President and Founder of Diller Digital, Tim Diller, pictured here.


 
1) What is your name and where are you currently located? 

My name is Tim Diller, and I live in Austin, TX with my wife Hannah and my dog Stella.  I have three adult children who are all out of the house now. 

2 ) How did you end up in engineering education? 

For me it has been a long, winding, and nearly closed-loop path.  My reference point is my father, who has spent his entire career in academia, combining research and teaching in Biomedical Engineering at The University of Texas at Austin.  From him and others in my family, I developed a high regard for education, and from an early age I aspired to professorship, and that vision plus a deep-seated curiosity about mechanical things (especially cars and airplanes) guided my steps through high school, my bachelor’s degree in Mechanical Engineering at The University of Texas, and into the first semester of a Master’s degree program at MIT, where I hit an academic wall, nearly failing out of the program.  On academic probation, I did some deep soul searching and realized that the math-heavy robotics program I had been pursuing was not a good fit for my natural talents and inclinations.  Instead, with some guidance, I pivoted to project-based courses in manufacturing and production systems design, where I thrived.  That led me to my first “real” job, at the Michelin Americas R&C Corporation. 

My time at Michelin helped me realize a few more things:  I love the collaborative team environment, I love learning about what people are doing in industry (working for a Tier 1 supplier in the automotive industry is great for that), and I love teaching (I had the opportunity to pick up, revamp, and deliver Michelin’s course on tire performance for vehicle handling during my time there).  I also found myself gravitating to software development projects and had my first exposure to Python at that time.  I spent 5 years there before a growing sense of “unfinished business” led me to return to graduate school for a doctoral degree at The University of Texas. 

During my doctoral program, I spent a lot of time instrument an engine and analyzing data on exhaust gases (if you’ve taken a class from me or read some of my other posts, you’ll notice I use a lot of automotive references). had many opportunities to teach, substituting for my advising professor from time to time, delivering guest lectures on tire performance for another professor in the department.  By the time I had finished my degree and was working as a postdoctoral researcher, this was a regular occurrence.  During those years, I also made the transition from MATLAB and because thoroughly hooked on Python.  At the same time that I was seeking employment in academia, my love of using software for scientific computing was growing.  Thus it was that during another round of academic hiring (this was during the period after the downturn in 2008), I found that Enthought was hiring, and right in Austin, where I was located. 

The opportunity at Enthought included working on interesting coding and consulting projects across a broad swath of industry, working collaboratively in small teams, and teaching a 40-hr, week-long class called Python for Scientists & Engineers, which I would eventually teach over 50 times for Enthought. 

When in 2023 Enthought retired their training department during a reorganization, I founded Diller Digital to provide continuity of service to the customers they had served for decades.  I get a lot of joy working with smart, motivated engineers, scientists, and analysts to help them increase their digital skills in scientific computing. 

3 ) How do you stay current with the latest advancements in engineering technology and industry practices? 

I read papers and a lot of tech-oriented news sources.  It’s a lot of fun for me to do that. I buy (paper!) books on the topics I teach about and mark them up, code the demo examples and play around. For example, at present (mid 2025) I’m in the middle of reading and coding my through Sebastian Raschka’s Build a Large Language Model (From Scratch).   From time to time I will do small consulting jobs to stay engaged.  And it turns out I learn a lot from my students when they ask good questions.  I’ve learned that often the best answer is “I don’t know, but let me look into that”, and I’ll do enough of a deep dive to get an answer the next day, but often I’ll keep going.  And sometimes I’ll incorporate new material into the course based on that.  Or post about it.

4 ) Can you describe your teaching philosophy and how it aligns with Diller Digital’s mission and values? 

I believe that technology should be used to elevate the value and dignity of humanity’s work.  I also believe that a thorough understanding of fundamental principles is critical as a solid foundation for future self-learning in scientific computing and solving problems with software.  Because of that, I emphasize lots of hands-on experience, getting students to do basic things by hand and on their own before teaching them how to automate work with higher-level tools.  Although they might articulate it a little differently, this is close to the philosophy Enthought used to develop the materials we deliver at Diller Digital.  Enthought was clearly formative for me in my approach to teaching. 

5) What engineering software and tools do you have experience with, and how do you incorporate them into your teaching? 

My day-to-day coding work takes place in two contexts.  In the classroom, I use Jupyter Lab, which is just about the perfect tool for that environment— it’s simple enough to get everybody on the same platform quickly, even if someone has never used it before.  For maintaining the demos, exercises, automation scripts, or any other more-involved coding work, I’ll use VS Code with the Flake8 and Diff extensions installed. 

In the past, I liked Sublime Text because of its multiple-cursor and block-editing capabilities, and before that I was a proud (and probably obnoxious) fan of emacs, which I’ll still use on occasion when logged into a server with text-only interface.  But for that environment, I have come to appreciate the lighter-weight nano editor, which is available in pretty much every text-only environment I use these days. 

In addition to Python and the scientific computing libraries we teach, I’ve spent substantial time with MATLAB, C, LabView, and Visual Basic.  My first programming language was BASIC for the TI-99/4A, whose CALL SPRITE was the key technology that let me write my own video games. 

6) How do you balance theoretical knowledge with practical, hands-on learning in your classes? 

I try to teach in a way that theory and practice complement each other.  I use theory to explain the “Why”, and practical, hands-on learning to explain the “How”.  For example, when it comes to talking about lists and sets, the theory is important for understanding why sets have such faster look-up times, but I make sure that knowledge is accessible by demonstrating and having students follow along with %timeit commands. 

7) Can you discuss your experience with project-based learning and how you guide students through the data analysis workflow? 

As I talked about earlier, in graduate school I really struggled with theory-heavy instruction and thrived in project-based classes.  In addition, I have watched my father (an engineer professor) develop a course for graduate students in designing inquiry-based instruction, which is closely related to project-based learning.  Over the decades, we have had many long discussions about and have developed a shared passion for the subject. 

With that in mind, I start each class by asking them what goals they have in mind and then probing as much as time allows.  On the one hand, I’m genuinely interested in finding out what people do, but on the other hand, getting a student to articulate their problem and the value they are bringing to their organization is critical to creating context for learning.  That was a big part of what I did as a consultant for Enthought, and I have carried that into the classroom. 

Once context is established, I assume that the students need to see and hear, to follow along on their own machine, then to do on their own and exercise recall before they’ll master a concept.  We do this at multiple scales, typically building up from using and mastering data types, moving on to useful code segments, and ending with some kind of realistic capstone project that ties everything together.  Each of Python Foundations, Data Analysis with Pandas, Machine Learning, and Deep Learning follow this arc, and in choosing exercises, we have worked hard to make sure the problem is simple enough to be tractable in the relatively short time we have in class yet also complex enough to provide experience that will be useful in day-to-day work. 

8) What strategies do you use to assess student understanding and provide constructive feedback on their work? 

We have lots of small “Give It A Try” exercises that are designed to cement understanding and surface any confusion.  In virtual courses, it’s a bit more of a challenge, because I have to rely on self-reporting, and not everyone likes to unmute and ask a question.  For in-person classes, I walk the room during such exercises and look for the pink background of error messages.  The barrier to asking questions in that environment is lower.  But in either case, I tend to get good questions. 

The Give It A Try exercises are conducive to having students share their code, so sometimes I’ll use someone’s answer to explain the solution and ask for peer-suggestions. 

9) What strategies do you use to communicate complex engineering concepts to students with varying levels of understanding? 

This is the real challenge, because there is always a good diversity of backgrounds and experience.  One thing I do is try to provide a meta-level discussion, letting students know how important the following concept is and whether understanding it is critical or they can ignore it if needed. 

Another thing I do is to treat every question like pure gold, no matter the level.  If they ask about something fundamental I’ve already explained 5 times, great!  Because in that case, they finally have the context to get it, and by asking the question, they’ve owned the concept.  And if they ask a real stumper, and I have to do homework afterward to figure it out, that’s great too. 

Finally, I use a lot of physical analogies.  Students who have been in my classes may recall I tend to use a lot of automotive analogies, referring to “popping the hood” or talking about engines, brakes, and clutches.  But if someone mentions something like owning a hobby farm during introductions, we’ll talk about examples from the farm during class. 
 

10) What is your favorite way to spend a Saturday? Favorite meal? 
My ideal Saturday starts in the yard, mowing, weeding, or trimming. Once that’s done and the house is clean, maybe my wife and I will take our dog Stella for a walk on the local greenbelt trail.  After that, I work in the shop, where I like to build or restore furniture, make picture frames, or turn a bowl on my lathe.  Double the points if one of my kids is working with me.  If I can fire up the smoker and keep some ribs (when there’s more time) or fish (if there’s less) going while I’m in the shop, that completes the perfect Saturday. 

Tim Diller with his Family

Thanks for your answers to these questions, Tim so we can get to know you better as one of our respected instructors.

“Popping the Hood” in Python

One man holds the hood of a car open while he and his friend look at the engine together.

Last weekend found me elbow-deep in the guts of my car, re-aligning the timing chain after replacing a cam sprocket. As I reflected on the joys of working on a car with only 4 cylinders and a relatively spacious engine bay, I found myself reflecting on one of the things I love best about the Python programming language — that is the ability to proverbially “pop the hood” and see what’s going on behind the abstractions. (With a background in Mechanical Engineering, car metaphors come naturally to me.)

As an Open Source, well-documented, scripted language, Python is already accessible. But there are some tools that let you get pretty deeply into the inner workings in case you want to understand how things work or to optimize performance.

Use the Source!

The first and easiest way to see what’s going on is to look at the inline help using Python’s built-in help() function, which displays the docstring using a pager. But I almost always prefer using the ? and ?? in IPython or Jupyter to display the just the docstring or all of the source code if available. For example consider the relatively simple parseaddr function from email.utils:

In [1]: import email

In [2]: email.utils.parseaddr?
Signature: parseaddr(addr, *, strict=True)
Docstring:
Parse addr into its constituent realname and email address parts.

Return a tuple of realname and email address, unless the parse fails, in
which case return a 2-tuple of ('', '').

If strict is True, use a strict parser which rejects malformed inputs.
File:      /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/email/utils.py
Type:      function

In our Python Foundations course, I can usually elicit some groans by encouraging my students to “Use the Source” with the ?? syntax, which displays the source code, if available:

In [3]: email.utils.parseaddr??
Signature: parseaddr(addr, *, strict=True)
Source:   
def parseaddr(addr, *, strict=True):
    """
    Parse addr into its constituent realname and email address parts.

    Return a tuple of realname and email address, unless the parse fails, in
    which case return a 2-tuple of ('', '').

    If strict is True, use a strict parser which rejects malformed inputs.
    """
    if not strict:
        addrs = _AddressList(addr).addresslist
        if not addrs:
            return ('', '')
        return addrs[0]

    if isinstance(addr, list):
        addr = addr[0]

    if not isinstance(addr, str):
        return ('', '')

    addr = _pre_parse_validation([addr])[0]
    addrs = _post_parse_validation(_AddressList(addr).addresslist)

    if not addrs or len(addrs) > 1:
        return ('', '')

    return addrs[0]
File:      /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/email/utils.py
Type:      function

Looking at the next-to-last line, you see there’s a path to the source code. That’s available programmatically in the module‘s .__file__ attribute, so you could open and print the contents if you want. If we do that for Python’s this module, we can expose a fun little Easter Egg.

In [4]: import this
# <output snipped - but try it for yourself and see what's there.>

In [5]: with open(this.__file__, 'r') as f:
   ...:     print(f.read())
   ...: 
s = """Gur Mra bs Clguba, ol Gvz Crgref

Ornhgvshy vf orggre guna htyl.
Rkcyvpvg vf orggre guna vzcyvpvg.
Fvzcyr vf orggre guna pbzcyrk.
Pbzcyrk vf orggre guna pbzcyvpngrq.
Syng vf orggre guna arfgrq.
Fcnefr vf orggre guna qrafr.
Ernqnovyvgl pbhagf.
Fcrpvny pnfrf nera'g fcrpvny rabhtu gb oernx gur ehyrf.
Nygubhtu cenpgvpnyvgl orngf chevgl.
Reebef fubhyq arire cnff fvyragyl.
Hayrff rkcyvpvgyl fvyraprq.
Va gur snpr bs nzovthvgl, ershfr gur grzcgngvba gb thrff.
Gurer fubhyq or bar-- naq cersrenoyl bayl bar --boivbhf jnl gb qb vg.
Nygubhtu gung jnl znl abg or boivbhf ng svefg hayrff lbh'er Qhgpu.
Abj vf orggre guna arire.
Nygubhtu arire vf bsgra orggre guna *evtug* abj.
Vs gur vzcyrzragngvba vf uneq gb rkcynva, vg'f n onq vqrn.
Vs gur vzcyrzragngvba vf rnfl gb rkcynva, vg znl or n tbbq vqrn.
Anzrfcnprf ner bar ubaxvat terng vqrn -- yrg'f qb zber bs gubfr!"""

d = {}
for c in (65, 97):
    for i in range(26):
        d[chr(i+c)] = chr((i+13) % 26 + c)

print("".join([d.get(c, c) for c in s]))

Another way to do this is to use the inspect module from Python’s standard library. Among many other useful functions is getsource which returns the source code:

In [6]: import inspect
In [7]: my_source_code_text = inspect.getsource(email.utils.parseaddr)

This works for libraries and functions that are written in Python, but there is a class of functions that are implemented in C (for the most popular version of Python, known as CPython) and called builtins. Source code is not available for those in the same way. The len function is an example:

In [8]: len??
Signature: len(obj, /)
Docstring: Return the number of items in a container.
Type:      builtin_function_or_method

For these functions, it takes a little more digging, but this is Open Source Software, so you can go to the Python source code on Github, and look in the module containing the builtins (called bltinmodule.c). Each of the builtin functions is defined there with the prefix builtin_, and the source code for len is at line 1866 (at least in Feb 2025 when I wrote this):

static PyObject *
builtin_len(PyObject *module, PyObject *obj)
/*[clinic end generated code: output=fa7a270d314dfb6c input=bc55598da9e9c9b5]*/
{
    Py_ssize_t res;

    res = PyObject_Size(obj);
    if (res < 0) {
        assert(PyErr_Occurred());
        return NULL;
    }
    return PyLong_FromSsize_t(res);
}

There you can see that most of the work is done by another function PyObject_Size(), but you get the idea, and now you know where to look.

Step by Step

To watch the Python interpreter step through the code a line at a time and explore code execution, you can use the Python Debugger pdb, or its tab-completed and syntax-colored cousin ipdb. These allow you to interact with the code as it runs and execute arbitrary code in the context of any frame of execution, including printing out the value of variables. They are the basis for most of the Python debuggers built in to IDEs like Spyder, PyCharm, or VS Code. Since they are best demonstrated live, and since we walk through their use in our Software Engineering for Scientists & Engineers class, I’ll leave it at that.

Inside the Engine

Like Java and Ruby, Python runs in a virtual machine, commonly known as the “Interpreter” or “runtime”. So in contrast to compiling code in, say, C, where the result is an executable object file consisting of system- and machine-level instructions that can be run as an application by your operating system, when you execute a script in Python, your code gets turned into bytecode. Bytecode is a set of instructions for the Python virtual machine. It’s what we would write if we were truly writing for the computer (see my comments on why you still need to learn programming).

But while it’s written for the virtual machine, it’s not entirely opaque, and it can sometimes be instructive to take a look. In my car metaphor, this is a bit like removing the valve cover and checking the timing marks inside. Usually we don’t have to worry about it, but it can be interesting to see what’s going on there, as I learned when producing and answer for a Stack Overflow question.

In the example below, we make a simple function add. The bytecode is visible in the add.__code__.co_code attribute, and we can disassemble it using the dis library and turn the bytecode into something slightly more friendly for human eyes:

In [9]: import dis
In [10]: def add(x, y):
    ...:     return x + y
    ...: 
In [11]: add.__code__.co_code
Out[11]: b'\x95\x00X\x01-\x00\x00\x00$\x00'
In [12]: dis.disassemble(add.__code__)
  1           RESUME                   0

  2           LOAD_FAST_LOAD_FAST      1 (x, y)
              BINARY_OP                0 (+)
              RETURN_VALUE

In the output of disassemble, the number in the first column is the line number in the source code. The middle column shows the bytecode instruction (see the docs for their meaning), and the right-hand side shows the arguments. For example in line 2, LOAD_FAST_LOAD_FAST pushes references to x and y to the stack, and the next line BINARY_OP executes the + operation on them.

Incidentally, if you’ve ever noticed files with the .pyc extension or folders called __pycache__ (which are full of .pyc files) in your project directory, that’s where Python stores (or caches) bytecode when a module is imported so that next time, the import is faster.

In Conclusion

There’s obviously a lot more to say about bytecodes, the execution stack, the memory heap, etc. But my goal here is not so much to give a lesson in computer science as to give an appreciation for the accessibility of the Python language to curious users. Much as I think it’s valuable to be able to pop the hood on your car and point to the engine, the oil dipstick, the brake fluid reservoir, and the air filter, I believe it’s valuable to understand some of what’s going on “under the hood” of the Python code you may be using for data analysis or other kinds of scientific computing.

You Still Need to Learn to Write Code in the Age of LLMs

Can we really delegate most or all of our coding tasks to LLMs? Should we tell our kids not to study computer science? What are the reasons we should still learn to write code in the age of LLMs?

There is a chorus of voices telling us that soon we will be able to hand all of our coding tasks over to an AI agent powered by an LLM. It will do all the tedious boring things for us, and we can focus on the important stuff. While the new generative AIs are amazing in their capabilities, I for one think we shouldn’t be so quick to dismiss the value of learning to code. Granted, I make my living teaching people to write software, so maybe I should call this “Why I don’t quit my job and tell everyone to use ChatGPT to write their code”, because I believe that learning to code is still necessary and good.

In early 2024 the founder and CEO of NVIDIA Jensen Huang participated in a discussion at the World Governments Forum that inspired countless blog posts and videos with titles like “Jensen Huang says kids shouldn’t learn to code!”. What he actually said is a bit different but the message is essentially the same [click here to watch for yourself, it’s the last 4 minutes or so of the interview]: “It’s our job to make computing technology such that nobody has to program … and that the programming language is human. Everybody in the world is now a programmer…“

Photo of Jensen Huang speaking at the 2024 World Governments Forum.

He suggests that instead of learning to code, he says we should focus on the Life Sciences because that’s the richest domain for discovery, and he thinks that developing nations that want to rise on the world stage should encourage their children to do the same. Oh, and he says we should build lots of infrastructure (with lots of NVIDIA chips of course).

There is a core part of his message I actually agree with. At Diller Digital, and at Enthought where our roots are, we have always believed it’s easier to add programming skills to a scientist, engineer, or other domain expert than it is to train a computer scientist in one of the hard sciences. That’s why if you’ve taken one of our courses, you’ve no doubt seen the graphic below, touting the scientific credentials of the development staff. And for that reason, I agree that becoming an expert in one of the natural sciences or engineering discipline is personally and socially valuable.

Image from the About Enthought slide in Enthought Academy course material showing that 85% of the developers have advanced degrees, and 70% hold a PhD.

At Enthought, almost no one was formally trained as a developer. Most of us (including me) studied some other domain (in my case it was Mechanical Engineering, the Thermal and Fluid Sciences, and Combustion in particular) but fell in love with writing software. And although there is a special place in my heart for the BASIC I learned on my brother’s TI/99 4A, or Pascal for the Macintosh 512k my Dad brought home to use for work, or C, which I self taught in High School and college, it was really Python that let me do useful stuff in engineering. Python has become a leading language for scientific computing, and a rich ecosystem has developed around the SciPy and NumPy packages and the SciPy conference. One of the main reasons is that it is pragmatic and easy to learn, and it is expressive for scientific computing.

And that brings me to my first beef with Huang’s message. While the idea of using “human language”, by which I believe he means “human language that we use to communicate with other humans” otherwise known as natural language, to write software has some appeal, it ignores the fact that we already use human language to program computers. If we were writing software using computer language, we’d be writing with 1s and 0s or hexadecimal codes. Although there are still corners of the world where specialists do that, it hasn’t been mainstream practice since the days of punch cards.

Image of human hands holding a stack of punch cards.  Image originally appears on IBM's web page describing the history of the punch card.

Modern computer languages like Python are designed to be expressive in the domain space, and they allow you to write code that is clear and unambiguous. For example, do you have any doubts about what is happening in this code snippet borrowed from the section on Naming Variables in our Software Engineering for Scientists & Engineers?

gold_watch_orders = 0
for employee in employee_list:
    gold_watch_orders += will_retire(employee.name)

Even a complete newcomer could see that we’re checking to see who’s about to retire, and we’re preparing to order gold watches. It is clearly for human consumption, but there are also decisions about data types and data structures that had to be made. The act of writing the code causes you to think about your problem more clearly. The language supports a way of thinking about the problem. When you give up learning the language, you inevitably give up learning a particular way of thinking.

This brings me to my second beef with the idea that we don’t need to learn programming. Using what Huang calls a “human language” in fact devolves pretty quickly into an exercise called “prompt engineering”, where the new skill is knowing how to precisely specify your goal to a generative model using a language that is not really designed for that. You end up needing to work through another layer of abstraction that doesn’t necessarily help. Or that is useful right up to the point where it isn’t, and then you’re stuck.

I often point my students to an article by Joel Spolsky called “The Law of Leaky Abstractions“, in which the author talks about “what computer scientists like to call an abstraction: a simplification of something much more complicated that is going on under the covers.” His point is that abstractions are useful and allow us to all sorts of amazing things, like send messages across the internet, or to our point, use a chat agent to write code. His central premise is there is no perfect abstraction.

All non-trivial abstractions, to some degree, are leaky.

Joel Spolsky

By that, he means that eventually the abstraction fails, and you are required to understand what’s going on beneath the abstraction to solve some tricky problem that eventually emerges. By the time he wrote the article in 2002, there was already a long history of code generation tools attempting to abstract away the complexity of getting a computer to do the thing you want to do. But inevitably, the abstraction fails, and to move forward you have to understand what’s going on behind the abstraction.

… the abstractions save us time working, but they don’t save us time learning.
And all this means that paradoxically, even as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder.

Joel Spolsky

For example, I’m grateful for the WYSIWYG editor WordPress provides for producing this blog post, but without understanding the underlying HTML tags it produces and the CSS it relies on, I’d be frustrated by some of the formatting problems I’ve had to solve. The WYSIWYG abstraction leaks, so I learn how HTML works and how to find the offending CSS class, and it makes solving the image alignment problem much much easier.

But it’s not only the utility of the tool. There’s a cognitive benefit to learning to code. In my life as a consultant for Enthought, and especially during my tenure as a Director of Digital Transformation Services, I would frequently recommend that Managers, and even sometimes Directors, take our Python Foundations for Scientists & Engineers, not because they needed to learn to code, but because they needed to learn how to think about what software can and can’t do. And with Diller Digital, the story is the same. Especially in the Machine Learning and Deep Learning classes, managers join because they want to know how it works, what’s hype and what’s real, and they want to know how to think about the class of problems those technologies address. People are learning to code as a way of learning how to think about problems.

The best advice I’ve heard says this:

Learn to code manually first, then use a tool to save time.

I’ll say in summary, the best reason to learn to code, especially for the scientist, engineers, and analysts who take our classes, is that you are learning how to solve problems in a clear, unambiguous way. And even more so, you learn how to think about a problem, what’s possible, and what’s realistic. Don’t give that up. See also this article by Nathan Anacone.

What do you think? Let me know in the comments.

Managing Pandas’ deprecation of the Series first() and last() methods.

Have you stumbled across this warning in your code after updating Pandas: “FutureWarning: last is deprecated and will be removed in a future version. Please create a mask and filter using `.loc` instead“? In this post, we’ll explore how that method works and how to replace it.

I’ve always loved the use-case driven nature of methods and functions in the Pandas library. Pandas is such a workhorse in scientific computing in Python, particularly when it comes to things like timeseries data and dealing with calendar-labeled data in particular. So it was with a touch of frustration and puzzlement that I discovered that the last() method had been deprecated, and its removal from Pandas’ Series and DataFrame types is planned. In the Data Analysis with Pandas` course, we used have an in-class exercise where we recommended getting the last 4 weeks’ data using something like this:

In [1]: import numpy as np
   ...: import pandas as pd
   ...: rng = np.random.default_rng(42)
   ...: measurements = pd.Series(
   ...:    data=np.cumsum(rng.choice([-1, 1], size=350)),
   ...:    index=pd.date_range(
   ...:        start="01/01/2025",
   ...:        freq="D",
   ...:        periods=350,
   ...:   ),
   ...:)
In [2]: measurements.last('1W')
<ipython-input-5-ec16e51fe7ce>:1: :1: FutureWarning: last is deprecated and will be removed in a future version. Please create a mask and filter using `.loc` instead
  measurements.last('1W')
Out[2]:
2025-12-15   -7
2025-12-16   -8
Freq: D, dtype: int64

This has the really useful behavior of selecting data based on where it falls in a calendar period. Thus the command above usefully returns the two elements from our Series that occur in the last calendar week, which begins (in ISO format) on Monday, Dec 15.

The deprecation warning says “FutureWarning: last is deprecated and will be removed in a future version. Please create a mask and filter using .loc instead.” Because .last() is a useful feature, I wanted to take a closer look to see if I could understand what’s going on and what the best way to replace it would be.

Poking into the code a bit, we can see that the .last() method is a convenience function that uses pd.tseries.frequencies.to_offset() to turn '1W', technically a designation of period, into an offset, which is subtracted from the last element of the DatetimeIndex, yielding the starting point for a slice on the index. From the definition of last:

 ...
    offset = to_offset(offset)

    start_date = self.index[-1] - offset
    start = self.index.searchsorted(start_date, side="right")
    return self.iloc[start:]

Note that side='right' in searchsorted() finds the first index greater than start_date. We could wrap all of this into an equivalent statement that yields no FutureWarning thus:

In [3]: start = measurement.index[-1] - to_offset('1W')
In [4]: measurement.loc[measurement.index > start]
Out[4]:
2025-12-15   -7
2025-12-16   -8
Freq: D, dtype: int64

There’s a better option, though, which is to use pd.DateOffset. It’s a top-level import, and it gives you control over when the week starts, which to_offset does not. Remember we are using ISO standards, so Monday is day 0:

In [5]: start = measurements.index[-1] - pd.DateOffset(weeks=1, weekday=0)
In [6]: measurements.loc[measurements.index > start]
Out[6]:
2025-12-15   -7
2025-12-16   -8
Freq: D, dtype: int64

Slicing also works, even if the start point doesn’t coincide with a location in the index. Mixed offset specifications are possible, too:

In [7]: measurements.loc[measurements.index[-1] - pd.DateOffset(days=1, hours=12):]
Out[7]:
2025-12-15   -7
2025-12-16   -8
Freq: D, dtype: int64

The strength of pd.DateOffset is that it is calendar aware, so you can specify the day of the month, for example:

In [8]: measurements.loc[measurements.index[-1] - pd.DateOffset(day=13):]
Out[8]:
2025-12-13   -7
2025-12-14   -6
2025-12-15   -7
2025-12-16   -8
Freq: D, dtype: int64

There’s also the non-calendar-aware pd.Timedelta you can use to count back a set time period without taking day-of-week or day-of-month into account. Note: as with all Pandas location-based slicing, it is endpoint inclusive, so 1 week yields 8 days’ measurements:

In [9]: measurements.loc[measurements.index[-1] - pd.Timedelta(weeks=1):]
Out[9]:
2025-12-09   -9
2025-12-10   -8
2025-12-11   -7
2025-12-12   -8
2025-12-13   -7
2025-12-14   -6
2025-12-15   -7
2025-12-16   -8
Freq: D, dtype: int64

You may have noticed I prefer slicing notation, whereas the deprecation message suggests using a mask array. There’s a performance advantage to using slicing, and the notation is more compact than the mask array but less so than the .last() method. In IPython or Jupyter, we can use %timeit to quantify the difference:

In [10]: %timeit measurements.loc[measurements.index[-1] - pd.DateOffset(day=13):]
45.7 μs ± 2.36 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [11]: %timeit measurements.last('4D')
56.3 μs ± 14.9 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [12]: %timeit measurements.loc[measurements.index >= measurements.index[-1] - pd.DateOffset(day=13)]
89.2 μs ± 6.31 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

After spending some time with git blame and the Pandas-dev source code repository, the reasons for the deprecation of the first and last methods make sense:

  • there is unexpected behavior when passing certain kinds of offsets
  • they don’t behave analogously to SeriesGroupBy.first() and SeriesGroupBy.last()
  • they don’t respect time zones properly

Hopefully this has been a useful exploration of pd.Series.last (and .first), their deprecation, and how to replace them in your code with the more-explicit and better-defined masks and slices. Happy Coding!

Arrays and Lists

I often get questions about the difference between the Python list and the NumPy ndarray, and we talk about this a lot in Python Foundations for Scientists & Engineers. But recently I had a question about the array object that is part of the Python language, and why don’t we use it for computations? Why is NumPy necessary if Python has an array data type? Well, this is a great question!

First of all, let’s take a quick look at the Python array. It’s found in the array standard library and usually imported as arr:

>>> import array as arr

Like a NumPy ndarray, it has a data type, and every element of the array has to conform to that type. Set the data type in in the first argument when creating the array. 'l' in this case specifies a 4-byte signed integer:

>>> a = arr.array('l', range(5))
>>> a
array('l', [0, 1, 2, 3, 4])

It has the same indexing and slicing behavior as a Python list, and we even see that it has the same “multiplication” behavior as lists: i.e. “multiplying” means repetition, not element-wise, arithmetic multiplication as with the NumPy ndarray.

With that in mind, computations look the same for arrays as for lists:

>>> b = arr.array(a.typecode, (v + 1 for v in a))
>>> b
array('l', [1, 2, 3, 4, 5])

Let’s use IPython’s %timeit magic to see how array computations fare in comparison with Python lists and NumPy ndarrays. For the list and array, we’ve specified a list comprehensions as the most efficient way to do the computations and a simple vector computation for the NumPy ndarray, and our test is to increment sequence of 10 integers:

In [1]: import array as arr
   ...: import numpy as np
   ...:
   ...: %timeit [val + 1 for val in list(range(10))]
   ...: %timeit arr.array('q', [val + 1 for val in arr.array('q', range(10))])
   ...: %timeit np.arange(10) + 1
279 ns ± 45.8 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
825 ns ± 5.22 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
1.03 μs ± 98.7 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

It’s surprising to note that list comprehension is fastest, beating array computation by a factor of 3 or so, and beating ndarray computation by a factor of 3.7! Suspecting that a 10-integer sequence may not be representative, let’s bump it up to 1000:

In [2]: %timeit [val + 1 for val in list(range(1000))]
   ...: %timeit arr.array('q', [val + 1 for val in arr.array('q', range(1000))])
   ...: %timeit np.arange(1000) + 1
28.4 μs ± 3.47 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
82.4 μs ± 9.05 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
1.82 μs ± 175 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

With more numbers to chew on, NumPy clearly pulls ahead, suggesting the overhead of creating an ndarray may dominate computation time for small arrays, but the actual computation time is relatively small. The Python array is still relatively inefficient compared to the Python list. Let’s make some helper functions to compare how long it takes to increment each value in lists, arrays, and NumPy ndarrays accross a broad range of different sizes. Each function will accept a list, array, or ndarray and return the time (in ms) required to increment each value by 1.

In [3]: from datetime import datetime
   ...:
   ...: def inc_list_by_one(l):
   ...:     start = datetime.now()
   ...:     res = [val + 1 for val in l]
   ...:     return (datetime.now() - start).total_seconds() * 1000
   ...:
   ...: def inc_array_by_one(a):
   ...:     start = datetime.now()
   ...:     res = arr.array(a.typecode, [val + 1 for val in a])
   ...:     return (datetime.now() - start).total_seconds() * 1000
   ...:
   ...: def inc_numpy_array_by_one(arr):
   ...:     start = datetime.now()
   ...:     res = arr + 1
   ...:     return (datetime.now() - start).total_seconds() * 1000

Now let’s record the computation times for a range of array sizes and plot them using matplotlib and seaborn:

In [14]: import matplotlib.pyplot as plt
    ...: import seaborn as sns
    ...:
    ...: times = []
    ...: for size in range(1, 10_000_001, 200_000):
    ...:     times.append((
    ...:         size,
    ...:         inc_list_by_one(list(range(size))),
    ...:         inc_array_by_one(arr.array('q', range(size))),
    ...:         inc_numpy_array_by_one(np.arange(size)),
    ...:     ))
    ...: times_arr = np.array(times)
    ...:
    ...: sns.regplot(x=times_arr[:, 0], y=times_arr[:, 1], label='Python list')
    ...: sns.regplot(x=times_arr[:, 0], y=times_arr[:, 2], label='Python array')
    ...: sns.regplot(x=times_arr[:, 0], y=times_arr[:, -1], label='Numpy array')
    ...: plt.xlabel('Array Size (num elements)')
    ...: plt.ylabel('Time to Increment by 1 (ms)')
    ...: plt.legend()
    ...: plt.show()
Computation time v array size

…and now it’s clear that computation times for Python lists and arrays scale with the size of the object much faster than NumPy ndarrays do. And it’s also clear that the Python array is no improvement for computations, static typing notwithstanding. Let’s quantify:

In [26]: print(f'Python list {np.polyfit(times_arr[:, 0], times_arr[:, 1], 1)[0]:.2e} (ms/element)')
    ...: print(f'Python array {np.polyfit(times_arr[:, 0], times_arr[:, 2], 1)[0]:.2e} (ms/element)')
    ...: print(f'NumPy ndarray {np.polyfit(times_arr[:, 0], times_arr[:, -1], 1)[0]:.2e} (ms/element)')
Python list 4.02e-05 (ms/element)
Python array 6.50e-05 (ms/element)
NumPy ndarray 2.01e-06 (ms/element)

So if we take the slope of those curves as a gauge of performance, Python arrays are roughly 40% slower in computation than lists, and the NumPy ndarray computations run in about 95% less time, on average for arrays that are not trivially small. This is because NumPy takes advantage of knowing the data type of each element and setting up efficient strides in memory for computations, which can be optimized at the operating system and hardware levels.

What, then, is the advantage of the Python array over a Python list? With helper functions similar to the ones above, we can return sys.getsizeof() sample list and array objects over the same range of sizes. Plotting the results, we see something interesting:

Memory consumption v array size

Memory consumption for both Python arrays and NumPy ndarrays scale exactly linearly with the number of elements (the q typcode specifies a signed integer with a minimum 8 bytes, which corresponds to the default int64 dytpe for integer NumPy ndarrays.) But notice how the Python list memory consumption increases stepwise. This is an implementation detail of the Python list type that allows it to quickly add elements without shifting memory after each append operation. And this is likely the explanation for why lists can be faster and more efficient than arrays. When a new value is written to an array, it likely often occurs that the some or all of the object needs to be shuffled in memory to find contiguous locations, but a list is optimized for appending, so a certain number of additions can be made before any reshuffling happens.

To test this, we can compare the computation times for an array computed with a list comprehension against an array that is copied and overwritten:

def inc_array_by_one_with_copy(a):
    start = datetime.now()
    res = copy(a)
    for i, val in enumerate(a):
        res[i] = val + 1
    return (datetime.now() - start).total_seconds() * 1000

Plotted, we see that the copy-fill approach is marginally faster, and there is less variation, likely due to fewer memory relocations.

Computation time v array size for arrays created with list comprehension and copy-fill strategies

In conclusion, it helps to understand that the array type is really intended as a Python wrapper around a C array, and there are some functions and codes that use it to efficiently return a Python object. But as for computations, I hope I’ve convinced you that the NumPy ndarray is the right type for computations in terms of both memory and computation efficiency. As for Python arrays, now you know what they are, and you can safely ignore them in favor of the ndarray for your scientific computations.