You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-31 - 17:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jul 31, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2012721260,0sleep20-21swapper/209:44:082
200429264,9sleep30-21swapper/307:06:373
215629164,9sleep00-21swapper/007:08:360
195128676,6sleep20-21swapper/207:05:552
200228564,6sleep10-21swapper/107:06:361
2352992928,1cyclictest32-21ksoftirqd/211:52:362
235599250,1cyclictest0-21swapper/309:25:093
234899250,24cyclictest0-21swapper/111:40:111
235599231,1cyclictest0-21swapper/311:24:213
235599230,0cyclictest0-21swapper/311:17:303
235299230,4cyclictest0-21swapper/210:06:522
235299230,22cyclictest0-21swapper/211:17:012
235299230,21cyclictest0-21swapper/211:34:072
235599220,22cyclictest0-21swapper/311:39:433
235299221,21cyclictest0-21swapper/209:18:152
234899220,1cyclictest0-21swapper/111:28:071
2343992220,1cyclictest0-21swapper/009:18:150
234399220,4cyclictest141rcu_preempt09:30:450
235299210,1cyclictest0-21swapper/212:39:202
2348992117,4cyclictest38150irq/131-snd_hda11:05:261
234399210,1cyclictest0-21swapper/011:35:290
234899201,18cyclictest13799-21sh11:16:501
235299190,1cyclictest0-21swapper/211:58:532
235299190,19cyclictest0-21swapper/212:06:422
234899199,6cyclictest22272-21munin-plugin-st12:40:001
2348991916,2cyclictest0-21swapper/109:50:321
2348991915,3cyclictest0-21swapper/110:39:561
2348991915,3cyclictest0-21swapper/110:39:551
234399190,3cyclictest0-21swapper/009:41:130
235599183,2cyclictest27378-21diskmemload09:23:193
2355991817,1cyclictest0-21swapper/309:50:143
235599180,1cyclictest0-21swapper/311:54:173
235299180,3cyclictest0-21swapper/209:47:512
2348991815,2cyclictest0-21swapper/112:26:391
2348991815,2cyclictest0-21swapper/112:26:381
234899180,2cyclictest0-21swapper/112:31:061
234899180,18cyclictest0-21swapper/110:59:561
234399180,1cyclictest0-21swapper/010:47:580
234399180,18cyclictest0-21swapper/011:34:070
2352991715,1cyclictest0-21swapper/210:55:142
235299170,0cyclictest0-21swapper/210:46:432
234899171,15cyclictest31654-21latency_hist10:30:001
234899170,3cyclictest0-21swapper/111:46:121
234899170,2cyclictest0-21swapper/110:19:281
234399170,1cyclictest0-21swapper/010:09:320
234399170,16cyclictest0-21swapper/010:51:340
235599162,9cyclictest0-21swapper/312:15:063
235299163,11cyclictest2557-21vmstat10:50:262
235299161,14cyclictest0-21swapper/211:03:442
235299160,2cyclictest0-21swapper/211:09:132
235299160,1cyclictest0-21swapper/209:24:262
235299160,15cyclictest0-21swapper/211:44:032
2348991615,1cyclictest0-21swapper/112:20:261
234899161,2cyclictest0-21swapper/109:14:291
234899161,2cyclictest0-21swapper/109:14:291
2348991611,4cyclictest14173-21fschecks_time09:40:161
234399162,4cyclictest141rcu_preempt10:20:150
234399160,15cyclictest12907-21sendmail09:40:000
234399160,15cyclictest0-21swapper/011:07:070
2355991513,1cyclictest0-21swapper/311:47:053
235599150,1cyclictest0-21swapper/310:54:073
2352991513,1cyclictest0-21swapper/208:45:152
235299150,7cyclictest0-21swapper/210:12:502
235299150,2cyclictest0-21swapper/209:59:392
2348991514,1cyclictest0-21swapper/109:25:071
2348991511,4cyclictest38150irq/131-snd_hda10:07:461
234899150,1cyclictest27378-21diskmemload09:59:361
234899150,15cyclictest0-21swapper/110:14:541
234899150,14cyclictest0-21swapper/111:10:171
234399150,5cyclictest141rcu_preempt12:07:350
234399150,15cyclictest0-21swapper/011:47:390
2355991413,1cyclictest0-21swapper/310:13:083
2355991412,1cyclictest0-21swapper/312:02:033
235599140,11cyclictest0-21swapper/310:31:083
235299141,6cyclictest30527-21kworker/u8:311:35:052
235299141,2cyclictest25007-21ssh12:21:232
235299140,14cyclictest0-21swapper/210:25:552
235299140,0cyclictest0-21swapper/210:18:162
234899142,1cyclictest20839-21chrt11:01:361
234899141,10cyclictest38150irq/131-snd_hda07:20:131
2348991410,3cyclictest38150irq/131-snd_hda07:55:131
2343991412,1cyclictest7543-21apache211:01:580
234399141,1cyclictest0-21swapper/008:50:140
234399140,1cyclictest0-21swapper/011:58:530
235599138,1cyclictest0-21swapper/311:04:143
235599134,8cyclictest28098-21ssh10:08:053
2355991312,1cyclictest0-21swapper/310:59:173
235599130,13cyclictest0-21swapper/312:22:363
235599130,13cyclictest0-21swapper/311:40:203
235599130,0cyclictest0-21swapper/309:56:123
235299139,2cyclictest0-21swapper/212:00:292
235299135,5cyclictest32-21ksoftirqd/208:55:152
235299133,9cyclictest7873-21runrttasks11:13:432
235299132,10cyclictest27378-21diskmemload09:13:372
235299132,10cyclictest27378-21diskmemload09:13:362
2352991310,2cyclictest5495-21ssh09:35:152
2352991310,2cyclictest28168-21grep08:00:272
235299130,12cyclictest0-21swapper/212:29:372
235299130,12cyclictest0-21swapper/212:29:372
235299130,11cyclictest0-21swapper/212:17:472
235299130,10cyclictest0-21swapper/210:35:102
235299130,10cyclictest0-21swapper/210:35:092
234899132,7cyclictest225-21systemd-journal08:55:011
2348991310,2cyclictest6410-21wc12:10:211
2348991310,2cyclictest3931-21cut11:50:141
2348991310,2cyclictest29147-21ssh09:49:411
2348991310,2cyclictest20805-21cut12:00:141
234399139,2cyclictest25586-21sed07:55:260
234399136,6cyclictest0-21swapper/008:45:130
234399131,3cyclictest0-21swapper/011:44:220
234399131,2cyclictest0-21swapper/008:25:130
2343991310,2cyclictest21257-21sendmail12:20:010
2343991310,2cyclictest11498-21sed07:25:270
234399130,3cyclictest0-21swapper/012:35:000
234399130,12cyclictest0-21swapper/009:11:400
234399130,12cyclictest0-21swapper/009:11:400
235599129,2cyclictest4003-21cat07:10:263
235599129,2cyclictest24522-21ssh09:46:023
235599129,2cyclictest19036-21cron07:45:003
235599129,2cyclictest15990-21wc07:35:273
235599129,2cyclictest15799-21cron10:40:003
235599129,2cyclictest15799-21cron10:40:003
235599129,2cyclictest14842-21sh10:20:003
235599129,2cyclictest13940-21df_abs09:40:153
235599129,2cyclictest12904-21turbostat12:15:003
235599129,1cyclictest0-21swapper/309:16:243
235599128,3cyclictest28684-21tail09:10:173
235599128,3cyclictest28684-21tail09:10:163
235599128,3cyclictest21570-21ssh10:43:013
235599128,2cyclictest31710-21ssh09:30:573
235599127,2cyclictest0-21swapper/310:00:143
235599124,4cyclictest225-21systemd-journal12:10:003
235599123,4cyclictest7095-21munin-run07:20:013
235599123,4cyclictest7095-21munin-run07:20:003
2355991211,1cyclictest0-21swapper/312:25:213
2355991211,1cyclictest0-21swapper/312:25:213
235599120,9cyclictest0-21swapper/311:11:293
235299129,2cyclictest7098-21awk07:20:012
235299129,2cyclictest7098-21awk07:20:002
235299129,2cyclictest4002-21sh09:34:382
235299129,2cyclictest3360-21ssh10:31:482
235299129,2cyclictest26949-21grep09:05:272
235299129,2cyclictest17813-21ssh10:40:222
235299129,2cyclictest12595-21grep08:35:282
235299129,2cyclictest10515-21chrt08:32:542
235299122,8cyclictest27378-21diskmemload09:50:492
2352991212,0cyclictest0-21swapper/208:10:002
2352991211,1cyclictest0-21swapper/211:45:322
2352991210,1cyclictest3814-21grep08:20:142
2352991210,1cyclictest10534-21grep07:25:182
235299120,5cyclictest0-21swapper/212:10:122
235299120,1cyclictest0-21swapper/212:33:212
235299120,1cyclictest0-21swapper/207:55:132
235299120,11cyclictest29126-21ssh11:25:442
235299120,11cyclictest23950-21ssh09:26:242
234899129,2cyclictest21096-21cut08:55:171
234899129,2cyclictest16597-21ssh10:40:141
234899129,2cyclictest15997-21tr08:45:151
234899126,5cyclictest7532-21apache210:54:551
234899123,5cyclictest225-21systemd-journal10:25:001
234899121,8cyclictest0-21swapper/108:30:141
2348991210,1cyclictest7071-21cron07:20:001
2348991210,1cyclictest7071-21cron07:20:001
234899120,1cyclictest0-21swapper/109:15:461
234899120,11cyclictest28791-21latency_hist09:30:011
234899120,11cyclictest22384-21cut07:50:191
234899120,11cyclictest12311-21cut07:30:151
234399129,2cyclictest7865-21ssh10:35:060
234399129,2cyclictest7865-21ssh10:35:060
234399129,2cyclictest3963-21ssh10:32:170
234399129,2cyclictest3639-21rm11:50:030
234399129,2cyclictest30779-21rm10:10:010
234399129,2cyclictest30059-21sh12:25:110
234399129,2cyclictest30059-21sh12:25:100
234399129,2cyclictest26541-21ssh11:25:040
234399129,2cyclictest2479-21cat08:15:220
234399129,2cyclictest13840-21df08:40:160
234399129,2cyclictest12899-21ssh10:56:560
234399129,2cyclictest11724-21cat11:15:270
234399127,2cyclictest9537-21sleep07:25:000
234399121,9cyclictest23732-21users12:20:240
2343991210,1cyclictest22079-21grep08:55:270
234399120,7cyclictest0-21swapper/011:20:060
234399120,7cyclictest0-21swapper/009:50:170
234399120,11cyclictest26912-21ssh10:26:230
235599119,1cyclictest15731-21cut12:35:213
235599119,1cyclictest10020-21cat08:30:263
235599118,2cyclictest9173-21taskset07:20:343
235599118,2cyclictest29348-21ssh11:25:543
235599118,2cyclictest27241-21ssh10:26:393
235599118,2cyclictest26563-21head11:05:173
235599118,2cyclictest21973-21cut07:50:163
235599118,2cyclictest19082-21cat08:50:193
235599118,2cyclictest11995-21ssh09:39:183
235599118,2cyclictest11985-21cat08:35:213
235599111,1cyclictest12998-21grep11:55:153
235299119,1cyclictest92362sleep207:21:222
235299119,1cyclictest3331-21tr07:10:182
235299119,1cyclictest22366-21sort07:50:182
235299119,1cyclictest18852-21sed08:50:172
235299119,1cyclictest17233-21cat07:40:152
235299119,1cyclictest14823-21tr07:35:162
235299119,1cyclictest14357-21sed08:40:202
235299118,2cyclictest7284-21tr08:25:212
235299118,2cyclictest22777-21ssh11:22:192
235299110,11cyclictest14441-21ssh10:00:132
235299110,10cyclictest16988-21sensors10:20:252
234899119,1cyclictest1557-21grep10:30:271
234899118,2cyclictest6841-21cut08:25:161
234899118,2cyclictest654-21tail08:10:271
234899118,2cyclictest5650-21ssh09:35:161
234899118,2cyclictest30453-21sed08:05:291
234899118,2cyclictest24983-21sh09:05:011
234899118,2cyclictest24413-21sh10:45:041
234899118,2cyclictest21121-21ssh11:21:021
234899118,2cyclictest19591-21tr07:45:141
234899118,2cyclictest18636-21grep07:40:271
234899118,2cyclictest16971-21sh10:00:521
234899118,2cyclictest16391-21sh09:22:111
234899118,2cyclictest16336-21ssh11:57:021
234899118,2cyclictest1612-21tr08:15:161
234899118,2cyclictest14823-21awk12:15:211
234899118,2cyclictest10056-21tr07:25:161
234899118,2cyclictest0-21swapper/112:05:291
234899118,1cyclictest8454-21ssh11:33:101
234899115,2cyclictest225-21systemd-journal08:00:231
2348991110,1cyclictest11986-21cut11:35:161
234399119,2cyclictest29207-21sh09:49:430
234399119,2cyclictest19779-21cat07:45:140
234399119,2cyclictest15674-21ntpq07:35:210
234399119,2cyclictest15374-21cat10:00:180
234399119,1cyclictest5917-21rm09:55:100
234399119,1cyclictest4269-21df08:20:160
234399119,1cyclictest2897-21cstates07:10:160
234399119,1cyclictest26738-21grep08:00:140
234399119,1cyclictest25499-21grep09:05:150
234399119,1cyclictest24809-21taskset09:03:020
234399119,1cyclictest23097-21ssh09:25:440
234399119,1cyclictest20477-21taskset12:00:040
234399119,1cyclictest190352sleep007:45:000
234399119,1cyclictest17273-21ssh12:36:050
234399119,1cyclictest16184-21taskset10:40:070
234399118,2cyclictest8418-21ssh12:11:310
234399118,2cyclictest31263-21cat08:10:000
234399118,2cyclictest21676-21basename07:50:140
234399118,2cyclictest1430-21ssh11:10:040
234399118,2cyclictest13036-21ssh09:20:180
234399111,1cyclictest0-21swapper/010:15:130
235599108,2cyclictest9500-21rm12:31:313
235599108,2cyclictest31449-21cut08:10:153
235599108,2cyclictest2233-21cat08:15:193
235599108,1cyclictest2489-21apt-config11:30:003
235599108,1cyclictest20818-21cat08:55:153
235599108,1cyclictest12564-21expr07:30:163
235599107,2cyclictest29412-21ssh10:47:393
235599107,2cyclictest28800-21chrt08:05:113
235599107,2cyclictest26903-21grep09:05:273
235599107,2cyclictest24073-21cut07:55:143
235599107,2cyclictest16515-21ssh10:20:203
235599104,4cyclictest24992-21munin-plugin-st09:05:003
235299108,2cyclictest20728-21wc07:45:262
235299108,1cyclictest248982sleep209:04:082
235299107,2cyclictest2418-21sort08:15:202
235299107,2cyclictest12899-21grep07:30:182
234899107,2cyclictest4423-21cut08:20:171
234899107,2cyclictest3022-21expr07:10:161
234899107,2cyclictest27035-21chrt09:06:031
234899107,2cyclictest14814-21sed08:40:271
234899107,2cyclictest11679-21tail08:35:181
234399109,1cyclictest9422-21cat08:30:180
234399109,1cyclictest29049-21cpuspeed_turbos08:05:170
234399109,1cyclictest13627-21cat07:30:260
234399109,1cyclictest11814-21if_enp0s31f608:35:190
234399108,1cyclictest5209-21expr07:15:150
234399108,1cyclictest5209-21expr07:15:140
23559998,1cyclictest28290-21seq08:01:353
23559998,1cyclictest21209-21taskset07:47:273
23559998,1cyclictest11023-21awk07:25:223
23559997,2cyclictest4741-21expr08:20:203
23559997,1cyclictest15861-21tr08:45:143
23559997,1cyclictest13434-21hostname08:40:143
23559996,2cyclictest7720-21cat08:25:263
23529996,2cyclictest28891-21cut08:05:162
23489997,1cyclictest14855-21cstates07:35:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional