"There’s more to software engineering than writing code"
There really isn't. There may be more to shipping products than writing code, but in organisations of any size that's been specialized into different, other, roles for decades.
We've always been a pretty apathetic nation. Whilst our close neighbours across the channel regularly blockade ports, roads with various farmyard vehicles, or chop of the heads of an entire ruling class. Everything seems like such an effort.
If your society has found itself in a situation where it needs to be rebuilt, it might be worth considering other forms of organisation, rather than a rebuild!
"Matrix Freedom, offered its “members” a simple and attractive prospect: freedom from debt. For fees of up to £8,000, it would sell them a set of instructions that would supposedly allow them to reclaim mortgage payments, write off tax bills and cancel fines. Customers were taught how to file legal claims in arcane language that referenced ancient law."
"In its best month, detailed in a Zoom meeting leaked online, Matrix apparently took in £500,000."
idk, the license tells me that there's a good reason why you shouldnt expect that.
Now, if you were paying for support from a vendor, that's a different matter.
"THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED."
"a database that keeps each record's details together can't take advantage of that: to read two columns, it has to read all five."
My understanding is that modern databases will take advantage of that if you create an index of those fields - ie it'll read from the index, not the records.
We generally refer to an index that holds the data it indexes at its leaves a "clustered index". But this only means that the tuples are organized according to the ordering of the index: you still generally store the entire record (all columns/attributes) together.
For example, if I cluster on my sorted primary key, that just means that the index can jump to the tuple with the correct primary key quickly (log time), but you will still end up reading* the entire tuple in the same way I explain in the blog. And if you only store a record ID that points to the other fields instead of the entire tuple at the leaves of the index, then that wouldn't be considered a clustered index.
I will note, however, that DBMSs should be smart enough to know that an aggregation on only an indexed column does not need to fetch the rest of the columns, only the data at the leaves of the index.
Parent was talking about covering indexes not clustered indexes. Postgresql doesn't have true clustered indexes, table is always heap and indexes are always separate.
A clustered index the table itself is an index and organized as such.
A covering index is separate from the table but contains all fields necessary for the result so the base table need not be accessed.
What would be nice is a standard way of marking columns as 'to be stored in indices only' and not in the main table. Add that to SQL and an RDBMS should have a much easier time doing the right thing.
One can argue each view of a database is merely a derivate of the WAL, and therefore every view is an index. The major benefit of columnar databases is the data stored being continuous in memory. Avoid all serialize/deserialize and control flow logic. Most core index data structures in relational DBs do not support this kind of tight vector view, Arrow[1]is a format that does.
Theoretically one can have a custom index extension which stores the data in a column-oriented manner, yet then one has major duplication, which is a problem for petabyte-scale databases, and is an even bigger problem in the Cloud.
KDB+ is the best implementation of a columnar DB in my opinion. Simple C, and one can understand how the data is stored down to the DRAM cells. Durability is just MMAP, which is fine as OLAP DBMSs tend not to be ACID.
In my personal opinion one shall learn from the best. The best in the world in massively parallel vectorized computation are the people at OpenAI and Anthropic, and the people who built PyTorch. Much of their work can be applied to analytical DBMSs.
Yes, this is the cleaner interpretation. More specifically, every index is a materialized view that is always updated atomically with the table itself.
I'm not sure, I have never seen that except when explicitly taking the index offline. To my mind it doesn't make sense for (active) indexes to be out of sync with the table they're referencing, but I can't vouch for the sanity of 100% of database implementations out there. Maybe read "should be" instead of "is".
Not really modern either.. Oracle/Sybase/SQL Server have been doing that already in the 90s.. including indexed views for aggregation
What they recently (and some of these were added in late 2000s) added is columnstore indexes... with compression and also in memory (Sybase Hana, SQL Server In Memory aka Hekaton, Oracle IM column store)
Index-only scans are possible when the index covers all involved columns and the query condition favors using the index. If you have an index on (col_a, col_b) but you run SELECT col_a WHERE col_b, you typically will not get an index-only scan because the query planner will decide against using the index in the first place, since it is ordered on col_a before col_b.
reply