You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 03:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 16, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12324210074,6sleep00-21swapper/019:08:380
1223928471,9sleep30-21swapper/319:07:333
1217128371,8sleep20-21swapper/219:06:392
1030228265,13sleep10-21swapper/119:05:091
1252399349,24cyclictest5541-21diskmemload22:04:203
1252399349,24cyclictest5541-21diskmemload22:04:203
1252399341,32cyclictest0-21swapper/300:35:143
1252399313,27cyclictest29158-21systemctl23:20:003
1252399312,28cyclictest5541-21diskmemload22:17:143
1251799310,3cyclictest225-21systemd-journal23:20:001
1252399300,27cyclictest26781-21grep22:20:153
1252399291,26cyclictest0-21swapper/321:38:113
1252399290,7cyclictest5541-21diskmemload23:53:243
1252399290,26cyclictest0-21swapper/322:14:473
1252399290,22cyclictest0-21swapper/321:50:193
1251799294,1cyclictest5541-21diskmemload23:59:141
1251799290,25cyclictest0-21swapper/121:54:091
1252399285,22cyclictest5541-21diskmemload23:28:113
1252399281,26cyclictest5541-21diskmemload23:36:223
1252399281,26cyclictest5541-21diskmemload23:36:223
1252399281,25cyclictest0-21swapper/320:15:143
12523992810,17cyclictest5541-21diskmemload22:31:233
1252399280,26cyclictest0-21swapper/321:55:513
1252399280,24cyclictest5541-21diskmemload22:46:383
1251799281,27cyclictest0-21swapper/100:18:431
1252399278,16cyclictest5541-21diskmemload00:23:333
1252399273,23cyclictest5541-21diskmemload21:40:483
1252399273,23cyclictest5541-21diskmemload00:05:163
1252399272,24cyclictest0-21swapper/321:30:143
12523992718,8cyclictest5541-21diskmemload00:18:433
1252399271,23cyclictest17806-21grep00:10:213
1252399270,26cyclictest0-21swapper/323:47:583
1252399270,21cyclictest0-21swapper/319:35:133
1251799273,3cyclictest18632-21apt-key22:35:011
1251799272,3cyclictest17257-21munin-run19:20:001
1252399269,15cyclictest32336-21ssh21:45:143
1252399264,21cyclictest5541-21diskmemload00:29:463
1252399263,22cyclictest455-21dbus-daemon19:20:003
1252399262,23cyclictest5541-21diskmemload23:42:133
1252399261,24cyclictest0-21swapper/323:33:413
1252399260,23cyclictest0-21swapper/321:15:143
1251799263,22cyclictest0-21swapper/121:10:051
1251799261,3cyclictest0-21swapper/122:12:491
1251799260,2cyclictest0-21swapper/121:29:181
1251799260,1cyclictest0-21swapper/100:05:151
1252399253,21cyclictest5541-21diskmemload23:55:123
1252399252,22cyclictest0-21swapper/323:01:573
1252399251,23cyclictest5541-21diskmemload00:30:193
1252399251,22cyclictest0-21swapper/322:40:133
1251799252,4cyclictest93-1kworker/1:1H19:40:001
1251799251,2cyclictest0-21swapper/100:30:181
1251799250,5cyclictest0-21swapper/123:27:391
1252399243,20cyclictest32002-21rm00:00:073
1252399241,21cyclictest5465-21ssh21:29:023
1251799242,3cyclictest5541-21diskmemload22:54:181
1251799241,2cyclictest26783-21cut22:20:161
1251799241,1cyclictest21900-21apt-get23:35:031
1251799241,1cyclictest21900-21apt-get23:35:021
1251299240,1cyclictest0-21swapper/000:35:410
1252399233,19cyclictest5541-21diskmemload21:10:043
1252399232,20cyclictest0-21swapper/320:35:133
1252399231,21cyclictest5541-21diskmemload23:21:143
12523992311,11cyclictest9861-21fw_packets20:10:163
12523992311,11cyclictest9861-21fw_packets20:10:153
1252399230,18cyclictest0-21swapper/320:45:253
1251799233,1cyclictest1482-21sed22:05:141
1251799233,18cyclictest25472-21awk22:20:001
1251799232,4cyclictest0-21swapper/100:20:351
1251799230,21cyclictest0-21swapper/123:41:171
1252399223,18cyclictest0-21swapper/322:05:153
1252399222,15cyclictest18778-21ssh23:13:263
1252399221,20cyclictest5541-21diskmemload21:20:133
1252399221,18cyclictest0-21swapper/322:55:143
1252399221,18cyclictest0-21swapper/321:00:013
1252399220,21cyclictest10093-21taskset23:08:103
1251799223,2cyclictest14542-21grep21:15:121
1251799221,20cyclictest0-21swapper/120:40:131
1252399211,20cyclictest0-21swapper/322:35:193
1252399211,19cyclictest9638-21rm22:29:533
1252399210,17cyclictest0-21swapper/322:50:453
1251799214,2cyclictest27495-21pmu-power20:45:231
1251799214,16cyclictest23190-21grep21:20:151
1251799213,2cyclictest0-21swapper/123:46:441
1251799211,4cyclictest25012-21aten2_r7power_e22:00:151
1251799211,4cyclictest25012-21aten2_r7power_e22:00:141
1251799210,4cyclictest0-21swapper/121:42:451
1251799210,16cyclictest0-21swapper/122:44:481
1252399202,16cyclictest1319-21unixbench_multi19:50:263
1252399201,18cyclictest18147-21fschecks_count19:20:163
1251799203,2cyclictest0-21swapper/122:59:511
1251799201,1cyclictest5541-21diskmemload23:52:591
1251799200,3cyclictest0-21swapper/119:55:121
1251799200,1cyclictest0-21swapper/100:10:171
1251799200,19cyclictest0-21swapper/120:10:121
1251799200,19cyclictest0-21swapper/120:10:111
1251799200,18cyclictest5541-21diskmemload23:20:221
1251299207,7cyclictest24612-21munin-run19:35:000
1252399191,17cyclictest31811-21sleep19:50:003
1252399191,17cyclictest0-21swapper/320:40:123
1251799194,2cyclictest16282-21cut21:35:191
1251799191,2cyclictest0-21swapper/100:00:071
1251799190,4cyclictest1317-21grep19:50:261
1251799190,4cyclictest0-21swapper/122:48:541
1251799190,2cyclictest0-21swapper/122:36:131
1251799190,1cyclictest0-21swapper/100:35:281
1251299193,2cyclictest141rcu_preempt23:00:130
1252399183,13cyclictest0-21swapper/320:50:173
1252399182,15cyclictest17864-21readlink20:25:223
1252399180,17cyclictest18872-21/usr/sbin/munin20:30:163
1252399180,17cyclictest0-21swapper/319:55:143
1252199180,1cyclictest0-21swapper/222:29:232
1251799188,5cyclictest19727-21sh23:33:191
1251799188,5cyclictest15837-21sh23:11:091
1251799184,1cyclictest5541-21diskmemload21:59:511
1251799181,2cyclictest31810-21munin-plugin-st19:50:011
1251799181,2cyclictest0-21swapper/121:45:161
1252399171,13cyclictest3177-21awk21:04:593
1252199178,1cyclictest0-21swapper/221:57:182
1252199175,11cyclictest0-21swapper/222:49:562
1251799177,4cyclictest21674-21kworker/1:100:25:141
1251299172,0cyclictest141rcu_preempt23:50:160
1252399169,6cyclictest4862-21expr20:00:143
1252399160,15cyclictest12861-21/usr/sbin/munin19:10:143
1252399160,14cyclictest0-21swapper/320:20:113
1252199160,2cyclictest0-21swapper/200:30:072
12517991614,1cyclictest0-21swapper/123:01:191
1251799161,2cyclictest5541-21diskmemload21:32:051
12517991610,4cyclictest5541-21diskmemload22:29:121
1251799160,1cyclictest0-21swapper/119:25:201
12512991613,2cyclictest0-21swapper/023:32:240
1251299160,16cyclictest0-21swapper/021:10:040
1252399159,5cyclictest5088-21grep21:05:293
1252399158,6cyclictest22440-21test19:30:013
1252399151,13cyclictest0-21swapper/319:40:123
1252399150,14cyclictest8557-21taskset20:05:453
1251799158,6cyclictest16753-21tr20:25:131
12517991510,4cyclictest29130-21grep20:50:151
1251799150,2cyclictest0-21swapper/123:06:091
1251299159,2cyclictest141rcu_preempt20:10:000
1252199149,3cyclictest7851-21df_inode21:30:162
12521991410,3cyclictest31128-21ls23:20:232
12521991410,3cyclictest23359-21wc23:55:002
1252199140,4cyclictest0-21swapper/221:14:042
1251799147,6cyclictest24609-21ls19:35:001
1251799147,6cyclictest21929-21cut20:35:171
1251799147,4cyclictest38150irq/131-snd_hda20:15:181
1251799142,3cyclictest0-21swapper/119:20:141
12517991412,1cyclictest12754-21apt-get19:10:131
1251799141,12cyclictest4424-21/usr/sbin/munin20:00:141
1251299144,5cyclictest225-21systemd-journal20:59:590
1251299143,6cyclictest2968-21munin-run22:44:590
12512991412,1cyclictest13-21ksoftirqd/022:20:000
1251299141,11cyclictest23672-21ssh23:55:100
12512991410,3cyclictest22310-21grep21:20:000
1251299140,1cyclictest0-21swapper/022:31:400
1251299140,14cyclictest0-21swapper/022:49:080
124472140,1sleep00-21swapper/021:32:590
226652130,2sleep00-21swapper/021:39:290
193052130,1sleep20-21swapper/222:54:442
1252399134,8cyclictest537-21snmpd19:28:553
1252199139,3cyclictest27491-21grep20:45:232
1252199139,3cyclictest22009-21ls19:30:002
1252199133,1cyclictest5541-21diskmemload00:02:572
12521991310,2cyclictest6429-21grep23:25:162
12521991310,2cyclictest23242-21grep19:30:182
12521991310,2cyclictest22903-21grep20:35:262
12521991310,2cyclictest21243-21sh00:12:262
12521991310,2cyclictest10439-21wc22:10:202
12521991310,2cyclictest10271-21uname20:10:182
12521991310,2cyclictest10271-21uname20:10:172
1252199130,12cyclictest5031-21grep21:05:292
1251799139,3cyclictest19648-21grep20:30:161
1251799137,5cyclictest31814-21tr20:55:161
1251799132,2cyclictest225-21systemd-journal20:21:021
1251799131,2cyclictest0-21swapper/121:00:201
1251799130,2cyclictest0-21swapper/120:05:251
1251299139,4cyclictest12887-21apache200:24:010
1251299139,3cyclictest23141-21tr21:20:150
1251299133,3cyclictest225-21systemd-journal00:34:590
1251299132,7cyclictest31799-21cron19:50:000
1251299131,6cyclictest30817-21munin-run20:54:590
12512991310,2cyclictest3384-21uname22:25:210
12512991310,2cyclictest21569-21tr20:35:140
12512991310,2cyclictest17245-21head20:25:160
12512991310,2cyclictest13-21ksoftirqd/023:26:530
1252199129,2cyclictest5541-21diskmemload00:05:172
1252199129,2cyclictest462-21grep23:40:262
1252199129,2cyclictest4118-21cron20:00:002
1252199129,2cyclictest32721-21sed20:55:272
1252199129,2cyclictest29289-21ssh23:00:152
1252199129,2cyclictest28924-21sed19:40:252
1252199129,2cyclictest27468-21sh21:41:402
1252199129,2cyclictest26896-21ssh22:00:282
1252199129,2cyclictest26896-21ssh22:00:272
1252199129,2cyclictest24132-21tr21:20:222
1252199129,2cyclictest16438-21tail21:35:182
1252199129,2cyclictest14335-21tr23:30:142
1252199129,2cyclictest13424-21grep19:10:172
1252199128,3cyclictest25910-21grep00:15:162
1252199128,3cyclictest16633-21grep20:25:132
1252199123,6cyclictest30166-21apache222:20:002
1251799129,2cyclictest3576-21wc21:05:171
1251799128,3cyclictest27284-21cut19:40:121
1251299129,2cyclictest7818-21cut23:45:160
1251299129,2cyclictest22089-21sed22:55:260
1251299129,2cyclictest21012-21sh20:35:000
1251299129,2cyclictest18170-21wc00:10:250
1251299129,2cyclictest18052-21chrt23:12:530
1251299129,2cyclictest14490-21df20:20:160
1251299129,2cyclictest12868-21chrt22:11:420
1251299126,5cyclictest8784-21perf22:10:000
12512991212,0cyclictest0-21swapper/021:44:090
1251299120,11cyclictest32339-21tr19:50:150
1251299120,0cyclictest0-21swapper/023:35:290
1251299120,0cyclictest0-21swapper/023:35:290
1252199119,1cyclictest27005-21head00:35:172
1252199119,1cyclictest0-21swapper/222:35:102
1252199118,2cyclictest8025-21grep20:05:212
1252199118,2cyclictest5675-21wc20:00:212
1252199118,2cyclictest4931-21ssh21:47:492
1252199118,2cyclictest4622-21ssh23:05:132
1252199118,2cyclictest4446-21ssh00:21:322
1252199118,2cyclictest30802-21taskset19:45:202
1252199118,2cyclictest28909-21ssh23:57:582
1252199118,2cyclictest27774-21cat22:20:252
1252199118,2cyclictest27748-21ssh23:18:562
1252199118,2cyclictest2659-21wc22:05:252
1252199118,2cyclictest25076-21swap20:40:242
1252199118,2cyclictest22529-21ssh23:35:112
1252199118,2cyclictest22529-21ssh23:35:102
1252199118,2cyclictest20623-21ls22:55:142
1252199118,2cyclictest19174-21tr20:30:142
1252199118,2cyclictest15263-21cut21:15:152
1252199118,2cyclictest12825-21ssh23:10:082
1252199118,2cyclictest12479-21ssh00:26:142
1252199118,2cyclictest10362-21ssh21:50:322
1252199115,5cyclictest28478-21/usr/sbin/munin22:40:182
1252199112,7cyclictest26968-21cron19:40:002
1252199110,3cyclictest0-21swapper/221:25:022
1252199110,10cyclictest17964-21ssh22:34:432
1251299119,1cyclictest25089-21grep19:35:160
1251299119,1cyclictest18287-21sort21:55:260
1251299118,2cyclictest920-21chrt23:21:420
1251299118,2cyclictest8846-21ssh23:07:110
1251299118,2cyclictest5061-21fschecks_time20:00:170
1251299118,2cyclictest4617-21pluginstate21:05:240
1251299118,2cyclictest3225-21ssh00:01:370
1251299118,2cyclictest32005-21sort21:25:160
1251299118,2cyclictest29120-21ssh22:21:210
1251299118,2cyclictest27066-21ssh22:00:320
1251299118,2cyclictest27066-21ssh22:00:320
1251299118,2cyclictest25702-21cat00:15:150
1251299118,2cyclictest24327-21sort20:40:160
1251299118,2cyclictest23527-21ssh23:15:370
1251299118,2cyclictest2221-21cat19:55:140
1251299118,2cyclictest20612-21grep19:25:160
1251299118,2cyclictest14223-21ssh21:53:340
1251299118,2cyclictest12611-21grep22:50:160
1251299118,2cyclictest10415-21cat00:25:180
1251299117,3cyclictest32040-21ssh21:45:000
1251299117,2cyclictest15398-21grep19:15:120
1252199108,1cyclictest7037-21ssh23:45:122
1252199108,1cyclictest17956-21expr19:20:142
1252199108,1cyclictest14962-21load20:20:182
1252199108,1cyclictest11687-21egrep20:15:152
1252199107,2cyclictest1759-21head21:00:152
1252199107,2cyclictest1448-21taskset19:51:092
1252199106,3cyclictest29390-21head20:50:172
1251299108,1cyclictest28191-21uname19:40:170
1251299108,1cyclictest26329-21cut20:45:150
1251299108,1cyclictest19249-21egrep22:35:150
1251299108,1cyclictest13144-21expr20:15:260
1251299107,2cyclictest9487-21cut00:05:220
1251299107,2cyclictest32410-21ssh23:40:210
1251299107,2cyclictest17762-21tr19:20:150
125219997,1cyclictest16934-21chrt19:16:132
125129996,2cyclictest1287-21grep21:00:130
125129996,2cyclictest12812-21cut19:10:140
125129990,3cyclictest0-21swapper/020:10:110
125129990,3cyclictest0-21swapper/020:10:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional