You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 20:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Jan 02, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
168152226163,51sleep20-21swapper/207:05:562
170542220192,19sleep30-21swapper/307:09:033
170782202165,25sleep10-21swapper/107:09:211
168902200164,24sleep00-21swapper/007:06:540
2362221770,5sleep01720699cyclictest07:20:230
2418421350,5sleep01720699cyclictest08:35:190
17222991101,107cyclictest0-21swapper/209:58:052
38882570,2sleep20-21swapper/211:30:192
313912560,1sleep30-21swapper/312:35:013
268472560,4sleep11721699cyclictest09:55:181
202812550,2sleep30-21swapper/310:55:123
295392510,2sleep00-21swapper/012:30:120
186462470,1sleep00-21swapper/012:05:120
17206993910,13cyclictest9718-21awk08:05:010
17222993524,7cyclictest18341-21kworker/2:212:28:242
1722299341,3cyclictest0-21swapper/211:21:342
1722299332,21cyclictest18341-21kworker/2:210:17:522
243032320,1sleep00-21swapper/009:50:160
1722299324,4cyclictest18341-21kworker/2:209:16:242
1722299324,4cyclictest18341-21kworker/2:209:16:242
24152310,8sleep011-21rcuc/011:25:390
1722299311,19cyclictest18341-21kworker/2:207:15:532
1720699311,15cyclictest10944-21munin-run10:35:010
1720699311,15cyclictest10944-21munin-run10:35:010
1722299301,16cyclictest18341-21kworker/2:209:45:522
1720699307,11cyclictest19132-21idleruntime-cro09:40:010
222662290,4sleep11721699cyclictest11:00:121
1722299291,13cyclictest18341-21kworker/2:210:04:462
1722299282,11cyclictest18341-21kworker/2:211:25:162
17222992815,2cyclictest18341-21kworker/2:212:19:442
1722299281,18cyclictest18341-21kworker/2:209:42:302
1722299281,12cyclictest18341-21kworker/2:209:52:422
1720699282,13cyclictest12525-21munin-run09:25:010
1722299277,8cyclictest18341-21kworker/2:211:13:442
1722299272,14cyclictest18341-21kworker/2:210:45:382
1721699279,15cyclictest32066-21cut08:55:141
1720699272,13cyclictest26965-21idleruntime-cro12:25:010
1722299260,12cyclictest18341-21kworker/2:209:00:142
1721699262,13cyclictest13139-21munin-run10:40:011
17216992611,13cyclictest3884-21kworker/u8:208:23:451
1720699261,13cyclictest24158-21awk11:05:010
17222992518,2cyclictest18341-21kworker/2:211:05:432
17222992513,3cyclictest18341-21kworker/2:211:57:102
17222992511,3cyclictest18341-21kworker/2:209:21:312
17222992511,3cyclictest18341-21kworker/2:208:50:112
17222992510,2cyclictest18341-21kworker/2:208:28:432
1722299250,5cyclictest18341-21kworker/2:211:43:582
1722299250,11cyclictest18341-21kworker/2:208:44:462
17216992512,11cyclictest20276-21kworker/u8:112:35:331
1721699251,19cyclictest0-21swapper/109:47:511
1721699251,14cyclictest0-21swapper/110:46:411
1720699254,13cyclictest15600-21sshd11:55:420
1720699253,13cyclictest0-21swapper/011:40:240
1722799241,6cyclictest0-21swapper/311:05:153
1722299249,2cyclictest18341-21kworker/2:208:15:182
1722299248,3cyclictest18341-21kworker/2:208:32:542
1722299246,2cyclictest30979-21kworker/2:012:34:062
1722299245,3cyclictest18341-21kworker/2:207:46:252
1722299242,13cyclictest18341-21kworker/2:207:13:432
1722299242,11cyclictest18341-21kworker/2:209:35:472
1722299242,10cyclictest18341-21kworker/2:208:00:512
17222992412,3cyclictest18341-21kworker/2:211:50:242
17222992412,2cyclictest18341-21kworker/2:210:30:162
17222992412,2cyclictest18341-21kworker/2:210:30:162
1722299241,16cyclictest18341-21kworker/2:212:24:082
1722299241,14cyclictest18341-21kworker/2:210:40:162
17222992411,3cyclictest18341-21kworker/2:212:06:262
17222992411,3cyclictest18341-21kworker/2:208:20:232
17216992411,11cyclictest12062-21kworker/u8:108:40:221
1722799231,5cyclictest0-21swapper/309:21:183
1722299239,2cyclictest18341-21kworker/2:207:43:392
1722299238,3cyclictest18341-21kworker/2:210:23:002
1722299237,3cyclictest18341-21kworker/2:207:35:032
1722299234,10cyclictest18341-21kworker/2:211:36:022
1722299232,18cyclictest18341-21kworker/2:209:25:122
1722299231,9cyclictest18341-21kworker/2:210:09:322
1722299231,4cyclictest0-21swapper/207:23:172
17222992311,3cyclictest18341-21kworker/2:211:16:522
17222992311,2cyclictest18341-21kworker/2:211:03:392
1722299231,12cyclictest18341-21kworker/2:209:30:582
17222992310,3cyclictest18341-21kworker/2:210:28:282
17222992310,3cyclictest18341-21kworker/2:208:56:112
1722299230,15cyclictest18341-21kworker/2:209:12:062
1722299230,10cyclictest18341-21kworker/2:210:35:482
1722299230,10cyclictest18341-21kworker/2:209:09:552
1721699232,4cyclictest25709-21munin-run09:55:011
1721699232,13cyclictest87250irq/129-enp1s0-12:25:141
1721699231,5cyclictest21198-21gltestperf12:10:161
1721699231,2cyclictest0-21swapper/107:26:201
17216992310,11cyclictest18961-21kworker/u8:110:40:321
1720699234,11cyclictest15916-21kworker/u8:212:17:470
1720699231,14cyclictest17922-21tr10:50:130
1720699231,11cyclictest28548-21perf11:15:020
1722799221,7cyclictest0-21swapper/311:52:593
1722799221,5cyclictest0-21swapper/311:32:373
1722299228,2cyclictest18341-21kworker/2:207:29:432
1722299227,3cyclictest18341-21kworker/2:207:52:352
1722299227,3cyclictest18341-21kworker/2:207:31:312
1722299223,3cyclictest30979-21kworker/2:012:39:042
17222992212,2cyclictest18341-21kworker/2:210:50:502
1722299221,12cyclictest18341-21kworker/2:208:08:152
1722299221,12cyclictest18341-21kworker/2:208:08:152
17222992210,2cyclictest18341-21kworker/2:212:12:282
1721699228,4cyclictest0-21swapper/111:15:271
1721699228,12cyclictest12062-21kworker/u8:108:37:131
1721699226,8cyclictest0-21swapper/109:35:391
1721699224,9cyclictest87250irq/129-enp1s0-10:20:141
1721699222,3cyclictest1581-21fschecks_time07:45:181
1721699222,18cyclictest0-21swapper/110:03:521
17216992218,3cyclictest0-21swapper/109:28:011
1721699221,3cyclictest11613-21ntp_states09:20:211
1721699221,2cyclictest0-21swapper/111:32:071
1721699221,19cyclictest0-21swapper/112:20:181
1721699221,19cyclictest0-21swapper/111:20:161
1721699221,18cyclictest8397-21hddtemp_smartct08:00:151
1721699221,18cyclictest1158-21grep11:25:151
1721699221,11cyclictest17247-21ntp_states12:00:221
1721699221,11cyclictest15610-21gltestperf09:30:161
1720699227,6cyclictest0-21swapper/010:58:490
1720699223,12cyclictest368-21kworker/u8:011:30:250
1720699222,12cyclictest0-21swapper/009:30:120
1720699221,5cyclictest0-21swapper/010:12:030
1722799219,4cyclictest11385-21NetworkManager11:47:593
1722799219,4cyclictest11385-21NetworkManager11:47:583
1722799211,6cyclictest0-21swapper/309:52:043
1722799211,5cyclictest30421-1kworker/u9:210:35:283
17227992115,3cyclictest27281-21tr12:25:123
1722799211,4cyclictest0-21swapper/311:57:433
17227992111,7cyclictest801-21snmpd11:03:313
1722799210,5cyclictest0-21swapper/310:02:513
1722299217,3cyclictest27510-21/usr/sbin/munin08:45:162
1722299217,3cyclictest18341-21kworker/2:208:35:052
1722299210,11cyclictest18341-21kworker/2:210:10:262
1721699219,3cyclictest411-21users07:40:261
1721699215,4cyclictest0-21swapper/111:39:191
17216992117,3cyclictest0-21swapper/112:07:571
1721699211,3cyclictest0-21swapper/108:06:221
1721699211,3cyclictest0-21swapper/108:06:221
1721699211,18cyclictest0-21swapper/109:09:401
1721699211,11cyclictest27510-21/usr/sbin/munin08:45:251
1721699211,11cyclictest19678-21df09:40:131
1721699211,10cyclictest18934-21proc_pri10:50:231
1721699211,10cyclictest13886-21cat11:55:011
1721699211,10cyclictest10506-21ls09:20:011
1721699211,10cyclictest10506-21ls09:20:011
1721699210,14cyclictest0-21swapper/111:45:241
1721699210,14cyclictest0-21swapper/111:45:241
1721699210,12cyclictest0-21swapper/110:07:361
1720699217,5cyclictest0-21swapper/009:11:110
1720699216,6cyclictest0-21swapper/009:48:030
1720699213,11cyclictest20276-21kworker/u8:112:35:350
1720699212,8cyclictest0-21swapper/010:46:420
1720699212,12cyclictest0-21swapper/011:08:260
1720699211,5cyclictest0-21swapper/012:01:140
1722799202,10cyclictest5191-21/usr/sbin/munin11:35:123
1722799202,10cyclictest23116-21/usr/sbin/munin08:35:183
1722799201,4cyclictest0-21swapper/312:00:153
1722799201,11cyclictest24980-21/usr/sbin/munin12:20:253
1722799201,11cyclictest14333-21/usr/sbin/munin08:15:193
1722299209,3cyclictest18341-21kworker/2:211:47:082
1722299209,3cyclictest18341-21kworker/2:211:47:082
1722299207,3cyclictest18341-21kworker/2:207:55:312
1722299207,2cyclictest18341-21kworker/2:212:04:202
1722299205,3cyclictest18341-21kworker/2:208:14:352
1721699209,3cyclictest0-21swapper/107:38:521
1721699208,2cyclictest20026-21awk10:55:111
1721699202,16cyclictest0-21swapper/108:30:021
1721699201,9cyclictest0-21swapper/111:07:211
17216992012,4cyclictest666-21fschecks_time10:10:151
1721699201,18cyclictest0-21swapper/110:28:391
1721699201,18cyclictest0-21swapper/108:53:541
1721699201,17cyclictest0-21swapper/108:25:121
1721699201,13cyclictest87250irq/129-enp1s0-10:18:151
1721699201,10cyclictest18891-21sed07:10:261
1720699209,3cyclictest1113-21/usr/sbin/munin07:45:290
1720699204,10cyclictest3884-21kworker/u8:209:16:450
1720699204,10cyclictest3884-21kworker/u8:209:16:450
1720699203,15cyclictest882-21perf07:45:010
1720699203,12cyclictest0-21swapper/009:25:500
1720699201,6cyclictest0-21swapper/008:07:030
1720699201,6cyclictest0-21swapper/008:07:030
17206992015,3cyclictest7432-21sed10:25:140
1720699201,13cyclictest32110-21kworker/u8:210:21:550
1720699201,12cyclictest21663-21ntpq12:10:220
1720699201,12cyclictest18961-21kworker/u8:110:41:150
1720699201,12cyclictest0-21swapper/011:46:530
1720699201,12cyclictest0-21swapper/011:46:530
1720699201,12cyclictest0-21swapper/010:36:020
1720699200,12cyclictest18961-21kworker/u8:109:43:300
1722799199,3cyclictest13455-21sort10:40:123
1722799195,6cyclictest0-21swapper/310:05:353
1722799191,10cyclictest8970-21/usr/sbin/munin10:30:253
1722799191,10cyclictest8970-21/usr/sbin/munin10:30:253
1722799191,10cyclictest31570-21/usr/sbin/munin12:35:163
1722299194,2cyclictest87250irq/129-enp1s0-10:56:302
1721699198,3cyclictest0-21swapper/112:30:181
1721699197,11cyclictest18961-21kworker/u8:111:10:321
1721699195,7cyclictest87250irq/129-enp1s0-10:35:011
1721699195,7cyclictest87250irq/129-enp1s0-10:35:011
1721699194,13cyclictest12062-21kworker/u8:108:19:511
1721699192,16cyclictest0-21swapper/107:59:471
17216991915,3cyclictest0-21swapper/107:18:051
1721699191,3cyclictest0-21swapper/108:10:081
1721699191,16cyclictest0-21swapper/109:14:021
1721699191,15cyclictest14790-21irqstats11:55:191
1721699190,18cyclictest0-21swapper/112:16:521
1721699190,17cyclictest0-21swapper/107:50:241
1720699197,4cyclictest22853-21sshd08:34:400
1720699194,10cyclictest18961-21kworker/u8:110:01:030
1720699191,9cyclictest0-21swapper/011:17:300
1720699191,12cyclictest0-21swapper/011:52:250
1720699191,12cyclictest0-21swapper/009:58:290
1720699190,12cyclictest15916-21kworker/u8:212:25:370
1720699190,11cyclictest0-21swapper/010:08:110
49412181,2sleep00-21swapper/007:50:330
304122181,2sleep30-21swapper/311:16:083
188972181,2sleep30-21swapper/309:37:233
1722799184,8cyclictest841-40dnf07:44:573
1722799182,6cyclictest25280-21perl07:25:153
1722799181,7cyclictest2149-21latency_hist10:15:013
1722799181,6cyclictest0-21swapper/311:28:203
1722799181,6cyclictest0-21swapper/311:11:453
1722799181,4cyclictest7386-21/usr/sbin/munin11:40:143
1722799181,4cyclictest21541-21/usr/sbin/munin09:45:183
17227991813,3cyclictest12050irq/123-ahci[0007:40:003
1722799181,15cyclictest0-21swapper/308:00:353
1722799181,11cyclictest0-21swapper/308:45:193
1722799181,11cyclictest0-21swapper/307:30:273
1721699187,9cyclictest368-21kworker/u8:011:40:121
17216991814,3cyclictest0-21swapper/107:30:101
17216991814,3cyclictest0-21swapper/107:25:001
1721699180,16cyclictest0-21swapper/109:00:071
1720699184,7cyclictest12127-21/usr/sbin/munin08:10:210
1720699182,4cyclictest1744-21crond08:45:010
1720699182,13cyclictest2724-21df_inode10:15:150
1720699181,9cyclictest5514-21sed09:05:250
1720699181,4cyclictest18961-21kworker/u8:111:36:110
1720699181,3cyclictest17581-21awk08:20:210
17206991813,4cyclictest12062-21kworker/u8:108:15:110
17206991811,4cyclictest0-21swapper/011:20:150
12782181,2sleep30-21swapper/308:57:223
307742171,2sleep00-21swapper/007:36:290
27612170,2sleep30-21swapper/307:45:473
212172171,2sleep30-21swapper/309:43:583
1722799174,4cyclictest0-21swapper/310:50:023
1722799172,7cyclictest6773-21/usr/sbin/munin10:25:143
1722799172,5cyclictest30421-1kworker/u9:212:10:133
1722799172,4cyclictest6711-21perl09:10:143
1722799172,2cyclictest87250irq/129-enp1s0-10:54:583
1722799171,7cyclictest20376-21ls07:15:133
17227991712,3cyclictest9193-21kernelversion09:15:183
17227991712,3cyclictest9193-21kernelversion09:15:183
17227991712,3cyclictest2380-21/usr/sbin/munin10:15:213
17227991712,3cyclictest14004-21sensors_temp09:25:253
1722799171,14cyclictest0-21swapper/307:51:403
17227991710,4cyclictest5190-21find09:05:223
17206991712,3cyclictest20311-21/usr/sbin/munin07:15:280
17206991711,3cyclictest28266-21timerandwakeup07:30:270
244232160,2sleep30-21swapper/312:15:563
1722799165,3cyclictest86950irq/126-enp1s0-08:20:223
1722799165,3cyclictest30421-1kworker/u9:211:23:413
1722799165,3cyclictest23491-21needreboot07:20:193
1722799165,3cyclictest12050irq/123-ahci[0008:15:013
1722799163,4cyclictest21729-21awk08:30:173
1722799161,6cyclictest801-21snmpd10:20:183
1722799161,6cyclictest25933-21/usr/sbin/munin09:55:143
1722799161,6cyclictest19517-21latency_hist07:15:013
1722799161,6cyclictest18390-21/usr/sbin/munin12:05:223
1722799161,4cyclictest16715-21sshd09:32:393
17227991612,3cyclictest0-21swapper/308:05:373
17227991612,3cyclictest0-21swapper/308:05:373
1722799161,14cyclictest0-21swapper/308:52:033
17227991610,4cyclictest25988-21smartctl08:40:163
1720699164,3cyclictest32672-21if_enp2s008:55:190
1720699164,3cyclictest19330-21gltestperf08:25:150
1720699162,12cyclictest17417-21/usr/sbin/munin07:10:270
1720699161,5cyclictest27584-21grep08:45:120
17206991612,3cyclictest0-21swapper/008:52:530
17206991611,3cyclictest6209-21hostname07:55:150
17206991611,3cyclictest2465-21basename09:00:170
1720699160,2cyclictest0-21swapper/007:25:230
1722799159,4cyclictest2115-21/usr/sbin/munin09:00:153
1722799155,3cyclictest86950irq/126-enp1s0-08:25:173
17227991510,3cyclictest5834-21expr07:55:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional