There was apparently a time when air travel was considered to be luxurious, and they had piano's in the lounges of the 747's. This all happened before my time because in my likely over 1000 flights (I traveled continuously for over a decade) there has been no glamor at all in the airline industry.
The industry was once regulated, then de-regulated, and then over the years all of the dominos fell on the "standard" airlines. In one of my stock funds for my nephews over here I bought Southwest Airlines stock just before 9/11 and today, more than eleven years later, that stock is trading at about the same level (before adjusting for dividends, to be fair). And of all the stocks, Southwest is the best performing - pretty much all the major US airlines went through the hair-raising process of bankruptcy at one point or another, meaning the shareholders (who held to the end) were wiped out.
Today, however, there seems to be some sort of stability to the industry. Over the last few years it seems that the planes I fly in get fuller and fuller, which is a testament to removing excess capacity from the system, as well as better practices. Once the planes are full it comes down to pricing, and the airlines have instituted a slew of new practices including charging for checked baggage and elimination of items like free meals on most flights which either raise revenue or reduce expenses.
The final element of all this is of course pricing. Since the airlines are often unable to differentiate on services (unless you are a heavy frequent flyer, in which case you have a built-in preference because miles on one airline are worth more than miles on any other airline), price wars are ruinous to profitability. Given that now most of the airlines have either emerged from bankruptcy or are in the process of the final shake-out (American), they seem to have adopted some sort of de-facto truce in the price wars because discounts on many flights seem to be less prevalent. Southwest could routinely be a source of lower cost airfares in the past, but even they seem to have raised their prices or perhaps they just limited the number of seats at the lowest price level so that it seems like the price is higher whenever I go to look for a flight.
Pricing now is so high on many routine flights that it becomes worthwhile to look at booking a limited business class flight (not unrestricted). Since the cabins are so full, it is difficult to compete for an upgrade unless you are a very high status flyer, and the completely cramped and packed conditions back in coach class are often correspondingly less tolerable. Most airlines now seem to charge an additional bump up to get a seat that provides legroom as well, so if you think you are paying a significantly better price you need to factor that into your comparison, as well. This would be for private travel only, since your company is likely not even going to consider paying for something like this, although they probably don't care if you upgrade with your miles.
If you are back in coach now and don't pay to get a coach seat that "doesn't suck" and don't pay or somehow get into a higher "group number", you are going to have to struggle to get your luggage onboard and then often get stuffed in a miserable middle seat. The entire experience in the back of the bus, now that all the seats seem to be filled, is correspondingly lousier.
The airlines can profit from this by instituting some sort of rational pricing system, where customers who are willing to pay a bit more (a few hundred dollars) get decent business class service including immediate onboarding, free drinks, getting off the plane first (it takes forever in the back when the plane is packed and the overhead bins are stuffed to the gills in order to avoid a checked luggage charge), can pay a bit more and then get this service. A few years ago it seemed that many of the front seats went vacant or were filled by frequent flyers upgrading (or being upgraded by the airline) at the gate. Today first / business class is completely full as is the rest of the plane and mostly rational pricing signals are being sent to customers. Whether those up front are paying or are high ranking frequent flyers is what the airline knows and not me but I'd suspect anecdotally that there are more "paying" up front customers today then there were a few years ago. I know I now think about business class, although it usually is out of the range of plausibility, especially when the price of a "regular" seat post upgrade to coach plus (or whatever the name is) already is starting at a high price.
It has taken probably 20 years for airlines to go from regulated to deregulated to (almost all) bankrupt and now reformed into a relatively small and efficient oligopoly. The airlines also seem to be sending rational price signals for the first time in my entire life. It has been a long strange trip for all of us.