You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 03:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Aug 14, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1499229366,8sleep30-21swapper/319:06:173
1495329265,9sleep20-21swapper/219:05:482
1518029077,9sleep00-21swapper/019:08:430
1522328568,13sleep10-21swapper/119:09:161
163222740,0sleep10-21swapper/122:31:101
1536999270,26cyclictest0-21swapper/223:21:032
1537199250,19cyclictest0-21swapper/322:02:123
15364992524,1cyclictest0-21swapper/122:28:541
15364992515,10cyclictest25-21ksoftirqd/123:19:531
321162240,0sleep30-21swapper/323:00:143
61502230,0sleep30-21swapper/323:42:043
1536499230,22cyclictest0-21swapper/100:35:251
1536499220,22cyclictest0-21swapper/121:37:571
1536999210,21cyclictest0-21swapper/223:34:272
1536999210,21cyclictest0-21swapper/200:19:042
1536499210,21cyclictest0-21swapper/123:11:311
1536499210,21cyclictest0-21swapper/122:51:341
1536499200,20cyclictest0-21swapper/123:57:351
1536499200,1cyclictest0-21swapper/121:11:461
1536099200,18cyclictest0-21swapper/022:00:120
1536999190,18cyclictest0-21swapper/222:41:152
1536999190,16cyclictest0-21swapper/200:28:162
1536499190,1cyclictest0-21swapper/122:47:331
1537199184,4cyclictest16528-31find00:07:563
1536999180,17cyclictest0-21swapper/223:42:182
1536099187,6cyclictest16341-21munin-run20:20:000
1536099183,11cyclictest24535-21ssh23:15:140
15360991815,3cyclictest13-21ksoftirqd/000:20:410
1536999170,17cyclictest0-21swapper/222:04:532
15369991614,2cyclictest32-21ksoftirqd/200:21:062
1536999160,16cyclictest0-21swapper/221:42:582
1536999160,15cyclictest0-21swapper/200:06:552
1536999160,12cyclictest0-21swapper/221:18:232
1536999160,0cyclictest0-21swapper/222:38:392
1536499164,5cyclictest24813-21perf21:20:011
1536499160,16cyclictest0-21swapper/122:09:371
1536499160,15cyclictest16528-31find00:07:511
1536499160,14cyclictest0-21swapper/122:24:361
1536099160,14cyclictest0-21swapper/021:26:180
1537199150,13cyclictest0-21swapper/300:20:133
1537199150,0cyclictest0-21swapper/322:35:253
1536999154,5cyclictest225-21systemd-journal23:05:012
1536999150,15cyclictest0-21swapper/223:35:352
1536999150,14cyclictest0-21swapper/222:33:422
1536999150,13cyclictest0-21swapper/200:14:072
15364991513,1cyclictest22731-21latency_hist19:25:001
15364991511,3cyclictest23713-21expr22:55:131
1536499150,13cyclictest0-21swapper/123:38:411
1536099156,3cyclictest25947-21awk21:40:000
291972141,1sleep08009-21diskmemload22:58:310
1537199147,5cyclictest0-21swapper/300:25:153
1537199147,2cyclictest29722-21fschecks_time22:20:183
1537199144,7cyclictest8009-21diskmemload21:30:013
15371991411,3cyclictest55650irq/127-enp2s0-00:37:543
1537199141,10cyclictest6119-21ls22:25:203
15371991410,3cyclictest5578-21grep22:05:263
15371991410,3cyclictest29151-21head23:55:243
15371991410,3cyclictest27283-21cron19:35:003
1537199140,13cyclictest3851-21fw_packets00:00:173
15369991413,1cyclictest0-21swapper/219:45:132
15369991410,3cyclictest5129-21grep21:00:242
15369991410,3cyclictest10159-21ntpq23:25:202
1536999140,1cyclictest0-21swapper/223:15:092
1536999140,14cyclictest0-21swapper/223:52:052
1536999140,13cyclictest30671-21tail19:40:172
1536999140,12cyclictest0-21swapper/223:57:202
1536499149,4cyclictest8511-21grep23:05:171
1536499147,6cyclictest19186-21apache221:57:531
15364991413,1cyclictest0-21swapper/120:40:131
15364991411,2cyclictest2909-21grep19:50:171
15364991411,2cyclictest25-21ksoftirqd/100:30:221
15364991411,2cyclictest25-21ksoftirqd/100:30:211
15364991411,2cyclictest23714-21wc22:35:261
15364991411,2cyclictest13574-21head00:25:191
15364991411,2cyclictest0-21swapper/100:23:441
15364991410,2cyclictest31006-21ssh22:40:161
15364991410,2cyclictest12578-21sed20:10:161
1536499140,14cyclictest0-21swapper/100:18:071
1536099140,1cyclictest0-21swapper/023:06:470
119442140,1sleep20-21swapper/222:10:132
232352130,0sleep30-21swapper/322:55:133
1537199139,4cyclictest19186-21apache221:38:553
1537199138,4cyclictest29685-21sshd21:41:253
1537199138,4cyclictest20593-21cut22:15:133
1537199138,3cyclictest0-21swapper/300:15:183
15371991310,3cyclictest0-21swapper/321:23:283
15371991310,2cyclictest9706-21tr20:05:113
15371991310,2cyclictest8373-21grep23:05:163
15371991310,2cyclictest18167-21sed23:30:173
1537199130,1cyclictest0-21swapper/323:50:133
1537199130,12cyclictest21594-21tr00:10:233
1536999139,3cyclictest4284-21sed21:45:282
1536999139,3cyclictest22047-21sed19:20:262
1536999139,3cyclictest20188-21ssh21:55:182
1536999137,2cyclictest225-21systemd-journal20:37:122
15369991310,2cyclictest6360-21grep22:25:222
15369991310,2cyclictest19325-21grep20:25:142
15369991310,2cyclictest13426-21sed20:10:262
1536999130,2cyclictest0-21swapper/221:10:032
1536999130,1cyclictest0-21swapper/221:05:142
1536499139,3cyclictest26281-21cat19:30:201
1536499139,2cyclictest24759-21tail20:35:171
1536499138,4cyclictest6478-21taskset19:56:261
15364991310,2cyclictest28859-21cut20:45:151
15364991310,2cyclictest2692-21cut23:40:161
1536499130,0cyclictest0-21swapper/122:14:391
1536099131,1cyclictest141rcu_preempt00:36:130
15360991310,2cyclictest26865-21ssh21:40:160
15360991310,2cyclictest21725-21basename20:30:140
15360991310,2cyclictest16413-21perf21:15:000
15360991310,2cyclictest14190-21grep22:30:160
1536099130,2cyclictest0-21swapper/023:03:590
1537199129,2cyclictest25675-21grep20:35:263
1537199129,2cyclictest22543-21sh19:25:003
1537199129,2cyclictest1836-21head20:55:153
1537199129,2cyclictest13732-21ssh21:12:583
1537199129,2cyclictest11609-21grep21:30:283
1537199128,3cyclictest4911-21tr19:55:133
1537199127,3cyclictest27522-21uname20:40:183
1537199127,2cyclictest21927-21perl00:30:173
1537199127,2cyclictest21927-21perl00:30:163
1537199126,2cyclictest225-21systemd-journal22:55:013
1537199123,3cyclictest15991-21/usr/sbin/munin23:10:193
1537199120,11cyclictest12412-21ssh21:50:273
1536999129,2cyclictest8657-21grep20:00:272
1536999129,2cyclictest30783-21wc22:20:262
1536999129,2cyclictest2910-21cut19:50:172
1536999129,2cyclictest26666-21grep20:40:132
1536999129,2cyclictest25951-21basename21:40:012
1536999129,2cyclictest20279-21wc19:20:012
1536999129,2cyclictest16318-21grep19:10:162
1536999129,2cyclictest16296-21ssh22:50:242
1536999129,2cyclictest1349-21tr20:55:132
1536999128,3cyclictest31907-21grep20:50:172
1536999128,3cyclictest29459-21chrt21:22:062
1536999128,3cyclictest11111-21grep21:50:182
1536999128,3cyclictest10485-21sed21:30:182
1536999126,2cyclictest225-21systemd-journal00:05:002
1536999125,6cyclictest55550irq/126-enp2s0-00:40:012
1536999120,12cyclictest0-21swapper/222:17:272
1536999120,12cyclictest0-21swapper/219:35:122
1536999120,10cyclictest0-21swapper/221:28:402
1536499129,2cyclictest32094-21uname20:50:181
1536499129,2cyclictest28184-21sort19:35:151
1536499129,2cyclictest27242-21head21:40:181
1536499129,2cyclictest23934-21grep19:25:211
1536499129,2cyclictest2035-21perf19:49:591
1536499129,2cyclictest13452-21chrt23:27:251
1536499129,2cyclictest11486-21wc21:30:281
1536499128,3cyclictest6764-21ssh23:23:441
1536499128,1cyclictest5478-21ssh23:03:431
1536499127,4cyclictest20137-21ssh23:50:211
1536499120,3cyclictest0-21swapper/121:48:201
1536499120,1cyclictest16064-21taskset20:17:101
1536499120,11cyclictest0-21swapper/122:20:001
1536099129,2cyclictest3941-21sed00:00:170
1536099129,2cyclictest29042-21tr20:45:150
1536099129,2cyclictest25248-21tr20:35:210
1536099129,2cyclictest21212-21cat00:10:190
1536099129,2cyclictest2058-21grep23:40:130
1536099128,3cyclictest27131-21grep20:40:160
1536099128,3cyclictest16376-21file00:07:440
1536099126,2cyclictest225-21systemd-journal19:45:010
1536099123,5cyclictest17354-21ssh23:30:130
1536099122,10cyclictest0-21swapper/022:22:100
1536099121,1cyclictest0-21swapper/022:39:060
1536099121,1cyclictest0-21swapper/022:08:160
15360991210,1cyclictest20472-21cut19:20:140
1536099120,1cyclictest0-21swapper/021:30:560
20862110,1sleep38009-21diskmemload21:45:013
1537199118,2cyclictest8258-21cat22:45:283
1537199118,2cyclictest7555-21tail21:05:273
1537199118,2cyclictest4035-21chrt23:21:373
1537199118,2cyclictest31088-21ntpq19:40:203
1537199118,2cyclictest28175-21tune2fs19:35:163
1537199118,2cyclictest27149-21ssh23:35:213
1537199118,2cyclictest26438-21ssh21:59:283
1537199118,2cyclictest25847-21ssh23:15:223
1537199118,2cyclictest22243-21cut20:30:163
1537199118,2cyclictest2198-21ssh22:42:063
1537199118,2cyclictest19509-21expr20:25:143
1537199118,2cyclictest18600-21chrt20:23:263
1537199118,2cyclictest17637-21cut21:15:163
1537199118,2cyclictest16328-21cron20:19:593
1537199118,2cyclictest16262-21cut19:10:163
1537199118,2cyclictest1595-21grep19:45:273
1537199118,2cyclictest15453-21ssh22:30:293
1537199118,2cyclictest13069-21cut20:10:223
1537199118,2cyclictest12522-21df22:10:163
1537199118,2cyclictest11562-21ssh23:25:573
1537199118,2cyclictest10643-21expr23:45:153
1536999119,1cyclictest22682-21wc00:30:232
1536999119,1cyclictest22682-21wc00:30:222
1536999118,2cyclictest7562-21ssh23:05:102
1536999118,2cyclictest5647-21wc19:55:182
1536999118,2cyclictest4422-21tail22:05:172
1536999118,2cyclictest26440-21ps19:30:222
1536999118,2cyclictest23462-21grep22:55:132
1536999118,2cyclictest23240-21ssh23:14:262
1536999118,2cyclictest23138-21expr19:25:152
1536999118,2cyclictest21901-21tr20:30:142
1536999118,2cyclictest18306-21head20:20:242
1536999118,2cyclictest14723-21df20:15:172
1536999117,3cyclictest7189-21ssh22:45:172
1536999113,4cyclictest225-21systemd-journal23:45:262
1536499118,2cyclictest7670-21taskset21:05:581
1536499118,2cyclictest7308-21tr20:00:161
1536499118,2cyclictest5063-21sed00:00:281
1536499118,2cyclictest30227-21tr19:40:141
1536499118,2cyclictest27375-21chrt22:00:011
1536499118,2cyclictest23754-21idleruntime-cro20:34:591
1536499118,2cyclictest20236-21cat00:10:131
1536499118,2cyclictest2018-21cat20:55:171
1536499118,2cyclictest19349-21ssh23:30:271
1536499118,2cyclictest1107-21ssh21:25:071
1536499115,2cyclictest17497-21ssh23:49:231
1536499110,3cyclictest8009-21diskmemload21:22:101
1536099119,1cyclictest30598-21tr22:40:150
1536099118,2cyclictest9459-21grep20:05:110
1536099118,2cyclictest8150-21wc21:10:010
1536099118,2cyclictest6583-21ssh22:25:240
1536099118,2cyclictest4668-21cut19:55:120
1536099118,2cyclictest3616-21ssh23:21:170
1536099118,2cyclictest32381-21grep20:50:210
1536099118,2cyclictest30399-21ssh23:56:210
1536099118,2cyclictest28171-21ssh00:15:040
1536099118,2cyclictest28043-21df19:35:140
1536099118,2cyclictest26804-21cat23:55:000
1536099118,2cyclictest26181-21expr23:35:130
1536099118,2cyclictest2521-21tr19:50:150
1536099118,2cyclictest21869-21ssh00:30:160
1536099118,2cyclictest21869-21ssh00:30:160
1536099118,2cyclictest20872-21ssh21:55:250
1536099118,2cyclictest20178-21sleep19:20:010
1536099118,2cyclictest14822-21tr22:50:140
1536099118,2cyclictest13708-21taskset21:51:090
1536099118,2cyclictest12030-21tr20:10:140
1536099118,2cyclictest11709-21ssh22:10:010
1536099118,2cyclictest10138-21ssh23:45:020
1536099117,3cyclictest17922-21grep23:10:250
1536099116,4cyclictest19186-21apache222:15:480
1536099113,4cyclictest225-21systemd-journal22:45:000
1536099113,4cyclictest225-21systemd-journal21:20:240
1536099111,2cyclictest141rcu_preempt21:15:020
1536099110,10cyclictest2182-21sh21:45:050
1536099110,10cyclictest0-21swapper/000:28:550
1537199108,1cyclictest5772sleep320:51:473
1537199108,1cyclictest3554-21cut21:00:123
1537199108,1cyclictest24228-21grep19:25:263
1537199108,1cyclictest200372sleep319:18:433
1537199107,2cyclictest29279-21sed20:45:173
1537199100,9cyclictest3709-21cat19:50:263
1536999108,1cyclictest28784-21apache_accesses20:45:122
1536999108,1cyclictest10519-21grep20:05:162
1536499108,1cyclictest19343-21grep20:25:141
1536499108,1cyclictest18041-21expr20:20:231
1536499108,1cyclictest16712-21cut19:10:191
1536499108,1cyclictest11194-21chrt20:05:441
1536499108,1cyclictest11127-21expr21:50:181
1536499107,2cyclictest3773-21tr21:00:131
1536499107,2cyclictest18272-21tr19:15:161
1536099108,1cyclictest9718-21head23:25:170
1536099108,1cyclictest3866-21expr21:00:120
1536099108,1cyclictest23068-21expr19:25:140
1536099107,2cyclictest32585-21tr19:45:140
1536099107,2cyclictest19351-21ls20:25:140
153719996,2cyclictest7702-21grep20:00:183
153609997,1cyclictest7117-21hostname20:00:150
153609997,1cyclictest184112sleep020:21:020
153609997,1cyclictest15805-21cat19:10:140
153609995,3cyclictest1379-21grep20:55:130
153609990,9cyclictest26580-21python19:30:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional