You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 21:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Mon Mar 16, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3310019912650,1251cyclictest3311462sleep307:10:023
3310019912000,1196cyclictest4146062sleep308:15:013
331001993050,302cyclictest542106-21turbostat09:54:563
331001992730,269cyclictest477078-21turbostat09:09:553
331001992601,255cyclictest730606-21turbostat12:04:563
54240721270,1sleep1321rcuc/109:50:121
72699260,5ntpd337825-21sshd07:15:020
72699240,11ntpd478227-21kworker/u16:0+nfsiod09:45:532
72699230,8ntpd483625-21diskmemload12:36:410
72699230,8ntpd483625-21diskmemload12:10:411
72699230,6ntpd750339-21ntpq12:10:200
72699230,5ntpd446172-21apt-get08:40:120
72699230,5ntpd446172-21apt-get08:40:110
72699230,5ntpd364709-21apt-get07:35:121
72699230,10ntpd518589-21ssh09:30:222
72699220,8ntpd772095-21ntpq12:25:192
72699220,8ntpd643933-21munin-run11:00:001
72699220,8ntpd506145-21ssh09:25:111
72699220,7ntpd708044-21ssh11:41:431
72699220,7ntpd526070-21awk09:35:240
72699220,7ntpd483625-21diskmemload12:22:572
72699220,6ntpd709251-21apt-get11:45:122
72699220,6ntpd540365-21ntp_states09:45:200
72699220,6ntpd483625-21diskmemload09:40:482
72699220,6ntpd400940-21ntpq08:00:190
72699220,6ntpd395960-21apt-get08:00:121
72699220,6ntpd383287-21latency_hist07:50:002
72699220,5ntpd389681-21apt-get07:55:121
72699220,5ntpd352238-21apt-get07:25:121
72699210,8ntpd483625-21diskmemload09:55:210
72699210,7ntpd657339-21ssh11:06:230
72699210,7ntpd578844-21cat10:15:001
72699210,7ntpd571629-21cut10:10:000
72699210,7ntpd562547-21sed10:00:241
72699210,7ntpd483625-21diskmemload10:56:302
72699210,6ntpd685461-21ntpq11:25:210
72699210,6ntpd351091-21ntpq07:20:210
72699210,5ntpd766425-21apt-get12:25:010
72699210,5ntpd644229-21apt-get11:00:110
72699210,5ntpd644229-21apt-get11:00:110
72699210,5ntpd638363-21irqstats10:55:140
72699210,5ntpd634964-21ntpq10:50:221
72699210,5ntpd620467-21ntpq10:40:202
72699210,5ntpd496483-21ntp_states09:15:200
72699210,5ntpd376082-21ntpq07:40:232
72699210,5ntpd363558-21ntpq07:30:200
72699210,4ntpd781487-21apt-get12:35:121
72699210,4ntpd738124-21ssh12:05:110
72699210,4ntpd452384-21apt-get08:45:121
72699200,7ntpd657892-21ssh11:08:251
72699200,6ntpd615160-21munin-run10:40:001
72699200,6ntpd396106-21df08:00:132
72699200,5ntpd728835-21ntp_states11:55:211
72699200,5ntpd723816-21df_inode11:55:120
72699200,5ntpd716802-21cat11:50:142
72699200,5ntpd672373-21updatedb.plocat11:18:371
72699200,5ntpd627661-21ntpq10:45:191
72699200,5ntpd599408-21ssh10:25:242
72699200,5ntpd584837-21ssh10:15:582
72699200,5ntpd562210-21ntp_states10:00:212
72699200,5ntpd547489-21ntpq09:50:200
72699200,5ntpd498744-21df_inode09:20:131
72699200,5ntpd483625-21diskmemload10:26:330
72699200,5ntpd471084-21apt-get09:00:112
72699200,5ntpd469902-21ntpq08:55:202
72699200,5ntpd432171-21ntpq08:25:202
72699200,5ntpd414407-21munin-run08:15:000
72699200,5ntpd388497-21awk07:50:200
72699200,5ntpd357296-21ntpq07:25:212
72699200,5ntpd329929-21ntpq07:05:200
72699200,4ntpd775602-21irqstats12:30:140
72699200,4ntpd757420-21ntpq12:15:202
72699200,4ntpd702010-21apt-get11:40:132
72699200,4ntpd694778-21apt-get11:35:112
72699200,4ntpd388446-21awk07:50:192
72699200,4ntpd377213-21apt-get07:45:111
72699200,4ntpd369823-21ntp_states07:35:220
72699200,4ntpd0-21swapper/210:32:492
72699190,8ntpd788-21mta-sts-daemon11:52:020
72699190,7ntpd463933-21sed08:50:242
72699190,6ntpd781115-21ssh12:34:501
72699190,6ntpd757317-21sed12:15:190
72699190,6ntpd581679-21cut10:15:160
72699190,6ntpd518461-21ntpq09:30:190
72699190,6ntpd382701-21unixbench_multi07:45:240
72699190,5ntpd714354-21ntpq11:45:200
72699190,5ntpd635538-21ssh10:50:382
72699190,5ntpd601159-21fschecks_count10:30:130
72699190,5ntpd593908-21taskset10:25:131
72699190,5ntpd496322-21grep09:15:171
72699190,5ntpd376038-21ntpq07:40:221
72699190,4ntpd679986-21ssh11:24:552
72699190,4ntpd648889-21tr11:00:172
72699190,4ntpd648889-21tr11:00:172
72699190,4ntpd614305-21ssh10:37:032
72699190,4ntpd586084-21apt-config10:20:001
72699190,4ntpd563972-21apt-get10:05:012
72699190,4ntpd498583-21apt-get09:20:120
72699190,4ntpd483625-21diskmemload11:33:080
72699190,4ntpd451207-21ntpq08:40:211
72699190,4ntpd451207-21ntpq08:40:201
72699190,4ntpd439915-21apt-get08:35:100
72699190,4ntpd388391-21grep07:50:181
72699190,4ntpd331191-21apt-get07:10:120
72699190,3ntpd707135-21ntpq11:40:220
72699190,3ntpd527888-21apt-get09:40:120
72699190,2ntpd0-21swapper/208:30:202
72699190,2ntpd0-21swapper/007:55:200
72699180,6ntpd576966-21ntpq10:10:220
72699180,6ntpd571182-21taskset10:08:561
72699180,6ntpd550611-21sed09:55:151
72699180,5ntpd715864-21ssh11:49:031
72699180,5ntpd699889-21ntpq11:35:200
72699180,5ntpd540314-21ssh09:45:211
72699180,5ntpd489048-21ntpq09:10:192
72699180,4ntpd787135-21runrttasks12:35:252
72699180,4ntpd773840-21ssh12:29:401
72699180,4ntpd738297-21ssh12:05:131
72699180,4ntpd737216-21seq12:02:472
72699180,4ntpd731802-21sed12:00:150
72699180,4ntpd728775-21ssh11:55:202
72699180,4ntpd657841-21ssh11:08:132
72699180,4ntpd629431-21ssh10:49:470
72699180,4ntpd620407-21tail10:40:181
72699180,4ntpd607679-21ssh10:34:201
72699180,4ntpd591708-21ssh10:20:222
72699180,4ntpd551809-21kthreadcore09:55:162
72699180,4ntpd525522-21ssh09:35:182
72699180,4ntpd512916-21ssh09:29:232
72699180,4ntpd489825-21ssh09:10:561
72699180,4ntpd483625-21diskmemload11:23:280
72699180,4ntpd451255-21ld08:40:222
72699180,4ntpd451255-21ld08:40:222
72699180,4ntpd426149-21smartctl08:20:242
72699180,4ntpd414804-21df_inode08:15:120
72699180,4ntpd344803-21ntpq07:15:192
72699180,3ntpd757464-21ssh12:15:211
72699180,3ntpd694242-21ssh11:34:132
72699180,3ntpd520631-21ssh09:35:121
72699180,2ntpd503798-21ssh09:20:212
72699180,2ntpd0-21swapper/210:47:302
72699180,1ntpd0-21swapper/008:50:110
6082412181,14sleep0608352-21/usr/sbin/munin10:35:120
33100199181,13cyclictest622321-21turbostat10:45:003
72699170,5ntpd681229-21irqrtprio11:25:162
72699170,4ntpd774178-21ssh12:30:022
72699170,4ntpd761192-21cat12:20:160
72699170,4ntpd743223-21ssh12:05:192
72699170,4ntpd686697-21sh11:27:551
72699170,4ntpd664685-21ssh11:11:412
72699170,4ntpd661689-21tr11:10:170
72699170,4ntpd511262-21gcc09:25:190
72699170,4ntpd496432-21ssh09:15:192
72699170,4ntpd480642-21grep09:05:171
72699170,4ntpd475625-21cut09:00:170
72699170,4ntpd425930-21ntpq08:20:210
72699170,4ntpd413302-21cat08:10:201
72699170,4ntpd363504-21/usr/sbin/munin07:30:192
72699170,4ntpd360269-21grep07:30:141
72699170,4ntpd346998-21cut07:20:151
72699170,3ntpd672484-21ssh11:18:560
72699170,3ntpd672400-21ssh11:18:372
72699170,3ntpd547811-21ssh09:50:242
72699170,3ntpd430787-21grep08:25:171
72699170,3ntpd376166-21sendmail_mailqu07:40:240
33100199171,11cyclictest665877-21chrt11:15:093
72699160,6ntpd804-21runrttasks07:17:161
72699160,4ntpd751100-21ssh12:11:002
72699160,4ntpd534091-21ssh09:42:151
72699160,4ntpd403562-21kthreadcore08:05:161
72699160,4ntpd402512-21if_eno108:05:142
72699160,3ntpd721618-21ssh11:50:211
72699160,3ntpd663820-21ssh11:10:221
72699160,3ntpd586813-21ssh10:20:141
72699160,3ntpd578225-21ssh10:13:142
72699160,3ntpd489214-21ps09:10:220
72699160,3ntpd452644-21fschecks_time08:45:142
72699160,3ntpd435712-21cut08:30:151
72699160,3ntpd390048-21ps07:55:152
72699160,3ntpd369773-21ntpq07:35:212
72699160,2ntpd0-21swapper/208:15:212
72699160,2ntpd0-21swapper/111:20:201
72699160,2ntpd0-21swapper/111:00:211
72699160,2ntpd0-21swapper/111:00:211
72699160,2ntpd0-21swapper/109:32:491
72699160,2ntpd0-21swapper/108:50:221
72699160,2ntpd0-21swapper/008:45:220
33100199168,4cyclictest395667-21turbostat07:59:593
331001991613,1cyclictest0-21swapper/308:13:093
33100199161,12cyclictest651469-21taskset11:05:123
33100199161,11cyclictest439213-21taskset08:30:243
72699150,3ntpd693764-21ssh11:32:211
72699150,3ntpd560035-21kthreadcore10:00:160
72699150,3ntpd434549-21irqrtprio08:30:140
72699150,2ntpd586988-21ssh10:20:150
72699150,2ntpd0-21swapper/207:20:212
72699150,2ntpd0-21swapper/207:05:202
72699150,2ntpd0-21swapper/112:00:201
72699150,2ntpd0-21swapper/108:35:201
72699150,2ntpd0-21swapper/108:20:221
72699150,2ntpd0-21swapper/108:15:201
72699150,2ntpd0-21swapper/010:40:120
72699150,2ntpd0-21swapper/009:05:220
72699150,2ntpd0-21swapper/008:55:200
72699150,2ntpd0-21swapper/007:25:220
72699150,1ntpd0-21swapper/112:20:061
72699150,1ntpd0-21swapper/010:50:120
33100199155,8cyclictest700598-21taskset11:35:573
33100199155,7cyclictest498590-21taskset09:20:123
33100199151,9cyclictest5789972sleep310:15:113
331001991513,1cyclictest0-21swapper/312:36:253
331001991513,1cyclictest0-21swapper/307:31:513
331001991513,0cyclictest0-21swapper/312:10:233
331001991512,1cyclictest0-21swapper/312:07:173
331001991512,1cyclictest0-21swapper/310:26:223
331001991512,1cyclictest0-21swapper/309:34:263
331001991512,1cyclictest0-21swapper/309:17:253
331001991512,1cyclictest0-21swapper/309:00:173
331001991512,1cyclictest0-21swapper/308:24:253
33100199151,11cyclictest781205-21taskset12:35:003
33100199151,11cyclictest642996-21taskset10:56:343
33100199151,10cyclictest764695-21taskset12:20:213
33100199151,10cyclictest562892-21taskset10:01:003
33100199151,10cyclictest488996-21taskset09:10:193
33100199151,10cyclictest344733-21taskset07:15:193
33100199150,11cyclictest470793-21turbostat09:00:003
72699140,3ntpd412497-21kthreadcore08:10:182
72699140,2ntpd694717-21ssh11:35:041
72699140,2ntpd0-21swapper/208:37:342
72699140,2ntpd0-21swapper/207:10:172
72699140,2ntpd0-21swapper/008:25:280
72699140,1ntpd0-21swapper/209:05:252
72699140,1ntpd0-21swapper/107:10:221
33100199145,6cyclictest6863252sleep311:26:293
33100199144,7cyclictest773024-21taskset12:26:393
33100199144,7cyclictest526613-21taskset09:36:313
33100199142,8cyclictest6730852sleep311:20:113
33100199141,9cyclictest5769332sleep310:10:223
33100199141,9cyclictest457987-21taskset08:45:343
331001991412,1cyclictest0-21swapper/307:50:103
331001991411,1cyclictest0-21swapper/311:50:093
331001991411,1cyclictest0-21swapper/311:30:243
331001991411,1cyclictest0-21swapper/310:50:033
331001991411,1cyclictest0-21swapper/310:49:163
331001991411,1cyclictest0-21swapper/308:51:503
331001991411,1cyclictest0-21swapper/307:45:353
331001991411,1cyclictest0-21swapper/307:22:473
33100199141,10cyclictest758228-21taskset12:16:293
33100199141,10cyclictest729375-21taskset11:55:323
33100199141,10cyclictest714448-21taskset11:45:233
33100199141,10cyclictest644226-21chrt11:00:113
33100199141,10cyclictest644226-21chrt11:00:103
33100199141,10cyclictest613963-21chrt10:35:423
33100199141,10cyclictest592534-21taskset10:21:403
33100199141,10cyclictest556068-21taskset09:58:303
33100199141,10cyclictest533896-21taskset09:41:323
33100199141,10cyclictest512092-21taskset09:26:173
33100199141,10cyclictest443874-21taskset08:35:173
33100199141,10cyclictest401788-21taskset08:03:393
33100199141,10cyclictest370589-21chrt07:38:433
33100199141,10cyclictest357615-21chrt07:25:253
33100199140,11cyclictest433085-21taskset08:29:563
33100199140,11cyclictest408145-21chrt08:09:563
72699130,2ntpd0-21swapper/107:05:161
72699130,1ntpd0-21swapper/109:00:201
72699130,1ntpd0-21swapper/008:05:200
33100199131,9cyclictest606124-21taskset10:30:213
33100199131,9cyclictest540960-21chrt09:45:493
33100199131,9cyclictest376941-21turbostat07:45:003
331001991312,0cyclictest0-21swapper/311:10:473
331001991312,0cyclictest0-21swapper/310:06:453
33100199130,11cyclictest446126-21taskset08:40:013
33100199130,11cyclictest446126-21taskset08:40:013
33100199130,10cyclictest701693-21sleep11:44:553
3306812130,3sleep30-21swapper/307:07:513
72699120,1ntpd0-21swapper/108:55:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional