You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 13:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon May 11, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1820429949,45sleep20-21swapper/219:07:072
1816729964,31sleep00-21swapper/019:06:370
1830928572,9sleep30-21swapper/319:08:293
1772328567,14sleep10-21swapper/119:05:261
327412560,0sleep00-21swapper/023:37:330
1853899291,23cyclictest16356-21ssh00:25:222
1853699260,0cyclictest0-21swapper/123:51:151
1853399260,26cyclictest0-21swapper/023:13:170
1853699250,22cyclictest0-21swapper/121:55:041
1853899230,0cyclictest0-21swapper/222:14:432
1853699230,22cyclictest0-21swapper/123:55:531
1853699230,22cyclictest0-21swapper/123:55:531
18533992222,0cyclictest0-21swapper/023:40:110
1853399221,3cyclictest141rcu_preempt22:06:090
1853399220,22cyclictest0-21swapper/022:49:170
1853899210,21cyclictest0-21swapper/223:47:342
1853699200,19cyclictest0-21swapper/119:30:261
18538991918,1cyclictest0-21swapper/223:00:142
1853399192,1cyclictest141rcu_preempt22:32:190
287582180,0chrt0-21swapper/122:37:031
1853899180,2cyclictest0-21swapper/200:23:492
1853699183,15cyclictest11943-21diskmemload23:25:131
1853699180,18cyclictest0-21swapper/123:41:111
1854399170,1cyclictest0-21swapper/300:07:463
1853899170,17cyclictest0-21swapper/221:21:112
1853899170,0cyclictest0-21swapper/222:15:372
1853899170,0cyclictest0-21swapper/200:30:232
1853399175,6cyclictest5594-21munin-run19:50:010
1853399170,15cyclictest0-21swapper/000:16:470
1854399162,1cyclictest0-21swapper/322:08:293
18543991615,1cyclictest0-21swapper/322:19:023
1854399160,12cyclictest0-21swapper/322:39:213
18538991615,1cyclictest0-21swapper/200:00:142
18538991615,1cyclictest0-21swapper/200:00:132
18538991610,2cyclictest31803-21grep22:00:172
1853899160,2cyclictest0-21swapper/221:50:442
1853899160,16cyclictest0-21swapper/222:28:332
1853899160,14cyclictest0-21swapper/221:48:292
18536991612,4cyclictest20960-21ssh22:13:211
1853699160,16cyclictest0-21swapper/122:45:111
1853699160,16cyclictest0-21swapper/100:32:321
1853699160,12cyclictest0-21swapper/120:35:131
1853399168,7cyclictest13340-21perf00:05:010
1853399168,7cyclictest13340-21perf00:05:000
1853399164,6cyclictest27553-21perf23:35:010
1853399161,9cyclictest17368-21/usr/sbin/munin22:50:130
1854399150,0cyclictest0-21swapper/323:46:033
1853899152,12cyclictest16489-21runrttasks23:08:442
1853899150,15cyclictest0-21swapper/223:11:322
1853699159,5cyclictest9228-21ntpq22:25:201
1853699153,6cyclictest225-21systemd-journal22:59:551
1853699150,14cyclictest0-21swapper/122:52:581
1853399151,13cyclictest0-21swapper/021:27:200
18533991511,2cyclictest3397-21tail20:50:160
1853399150,2cyclictest141rcu_preempt21:47:080
117472150,0sleep30-21swapper/322:45:583
18538991414,0cyclictest0-21swapper/222:30:232
18538991412,1cyclictest0-21swapper/223:23:252
18538991411,2cyclictest32-21ksoftirqd/222:07:222
1853899140,12cyclictest26363-21cut19:25:152
1853699149,4cyclictest21191-21cut19:15:131
1853699149,4cyclictest14837-21ssh00:25:081
1853699146,1cyclictest20313-21df_inode23:30:161
1853699142,11cyclictest32005-21/usr/sbin/munin00:15:211
18536991411,2cyclictest13010-21ssh21:10:141
1853399143,7cyclictest0-21swapper/021:58:540
18533991411,2cyclictest1011-21grep00:35:250
18533991411,2cyclictest0-21swapper/021:39:400
1853399140,1cyclictest0-21swapper/023:57:030
1853399140,1cyclictest0-21swapper/023:57:020
1853399140,1cyclictest0-21swapper/023:50:180
1853399140,14cyclictest0-21swapper/023:26:190
1854399139,3cyclictest31841-21sed19:35:173
1854399139,3cyclictest28565-21basename23:55:013
1854399138,3cyclictest22973-21tr21:35:193
1854399136,4cyclictest475-21in:imuxsock20:36:393
18543991310,2cyclictest8515-21grep00:20:253
18543991310,2cyclictest5781-21tr00:00:153
18543991310,2cyclictest5781-21tr00:00:153
18543991310,2cyclictest5488-21sh21:45:003
18543991310,2cyclictest4957-21sed23:20:263
18543991310,2cyclictest30438-21cron22:00:013
18543991310,2cyclictest15232-21grep00:25:163
18543991310,2cyclictest1117-21ssh22:20:263
18538991312,1cyclictest0-21swapper/221:28:322
18538991310,2cyclictest949-21wc19:40:002
18538991310,2cyclictest736-21grep20:45:172
18538991310,2cyclictest21765-21ssh21:35:132
18538991310,2cyclictest18635-21grep20:15:172
18538991310,2cyclictest11215-21grep20:00:172
1853899130,12cyclictest27945-21grep20:35:142
1853699139,3cyclictest22456-21grep00:10:131
1853699138,4cyclictest30869-21ssh22:00:091
1853699138,4cyclictest1818-21ssh21:42:081
18536991313,0cyclictest0-21swapper/123:05:111
18536991310,3cyclictest2515-21apache221:24:051
18536991310,2cyclictest5773-21sort21:25:151
18536991310,2cyclictest23765-21sort20:25:211
18536991310,2cyclictest22652-21wc20:25:011
18536991310,2cyclictest1740-21grep20:45:281
18536991310,2cyclictest15013-21wc00:05:191
1853699130,13cyclictest11943-21diskmemload00:03:441
1853699130,13cyclictest11943-21diskmemload00:03:431
1853399139,3cyclictest10833-21taskset23:05:170
1853399139,3cyclictest10813-21cron23:25:000
1853399135,4cyclictest225-21systemd-journal22:22:010
18533991310,2cyclictest27045-21sshd20:30:260
18533991310,2cyclictest23625-21grep00:10:200
18533991310,2cyclictest20459-21grep20:20:130
18533991310,2cyclictest20239-21grep20:20:000
1853399130,10cyclictest0-21swapper/020:55:130
1854399129,2cyclictest8310-21chrt21:45:393
1854399129,2cyclictest8241-21sed21:00:173
1854399129,2cyclictest5286-21tr20:55:143
1854399129,2cyclictest4212-21grep19:45:173
1854399129,2cyclictest405-21grep22:00:273
1854399129,2cyclictest32190-21cron20:45:013
1854399129,2cyclictest30561-21grep21:20:213
1854399129,2cyclictest24294-21sed19:20:163
1854399129,2cyclictest23987-21ssh00:10:233
1854399129,2cyclictest23430-21grep00:30:143
1854399129,2cyclictest2179-21cat20:50:003
1854399129,2cyclictest11019-21ssh23:05:173
1854399129,2cyclictest1045-21sed00:35:263
1854399128,3cyclictest20589-21grep20:20:143
1854399128,3cyclictest12719-21cron20:05:003
1854399127,5cyclictest2488-21apache221:15:503
1854399127,3cyclictest28125-21wc23:15:213
1853899129,2cyclictest7935-21ssh22:44:332
1853899129,2cyclictest5112-21ssh00:19:142
1853899129,2cyclictest4790-21cron20:55:002
1853899129,2cyclictest32411-21head00:35:172
1853899129,2cyclictest30769-21sort21:40:162
1853899129,2cyclictest28920-21grep19:30:152
1853899129,2cyclictest26107-21seq21:56:352
1853899129,2cyclictest21729-21sed21:15:182
1853899129,2cyclictest21548-21ls20:20:212
1853899129,2cyclictest19510-21grep19:10:172
1853899129,2cyclictest19156-21cron23:30:012
1853899129,2cyclictest13642-21tr21:30:162
1853899129,2cyclictest1302-21grep19:40:142
1853899127,4cyclictest28255-21egrep23:35:162
1853899120,5cyclictest19514-21sshd00:08:162
1853899120,2cyclictest0-21swapper/221:13:502
1853899120,1cyclictest0-21swapper/220:40:132
1853699129,2cyclictest29581-21tr21:40:001
1853699129,2cyclictest20615-21head23:10:271
1853699129,2cyclictest19507-21head19:10:171
1853699129,2cyclictest16126-21cut20:10:171
1853699129,2cyclictest14891-21ls21:30:251
1853699129,2cyclictest13605-21cat23:45:211
1853699128,3cyclictest3848-21sed23:00:261
1853699127,4cyclictest8676-21ssh00:20:291
1853699127,4cyclictest24829-21grep19:20:211
1853699124,7cyclictest6985-21ssh22:05:141
1853699120,6cyclictest225-21systemd-journal21:46:531
1853399129,2cyclictest947-21cat19:40:010
1853399129,2cyclictest9327-21grep19:55:200
1853399129,2cyclictest9048-21cat22:25:180
1853399129,2cyclictest31086-21basename00:35:010
1853399129,2cyclictest17224-21head22:10:270
1853399129,2cyclictest15108-21munin-run20:10:000
1853399129,2cyclictest13976-21taskset21:30:160
1853399128,3cyclictest10076-21tr21:05:160
18533991211,1cyclictest0-21swapper/019:21:540
1853399120,2cyclictest0-21swapper/021:51:480
1853399120,10cyclictest0-21swapper/000:27:200
1853399120,0cyclictest0-21swapper/000:23:150
1854399118,2cyclictest9363-21sh23:43:033
1854399118,2cyclictest8121-21cut19:55:003
1854399118,2cyclictest7811-21ssh22:25:073
1854399118,2cyclictest6583-21gzip21:25:213
1854399118,2cyclictest31711-21cat00:15:183
1854399118,2cyclictest30498-21seq19:31:183
1854399118,2cyclictest28391-21ssh22:55:443
1854399118,2cyclictest23508-21ssh22:53:383
1854399118,2cyclictest21388-21cut19:15:143
1854399118,2cyclictest20880-21wc23:30:193
1854399118,2cyclictest20316-21ssh22:32:003
1854399118,2cyclictest19303-21ssh23:10:163
1854399118,2cyclictest1903-21ssh23:00:133
1854399118,2cyclictest15989-21ssh21:30:533
1854399118,2cyclictest15428-21ssh21:11:113
1854399118,2cyclictest1491-21ssh23:38:283
1854399118,2cyclictest14686-21ssh21:50:153
1854399118,2cyclictest14631-21grep20:05:273
1854399118,2cyclictest13619-21ssh23:25:403
1854399117,3cyclictest29369-21aten2_r7power_p23:55:133
1854399117,3cyclictest29369-21aten2_r7power_p23:55:133
1853899118,2cyclictest7535-21grep19:50:242
1853899118,2cyclictest28709-21ssh23:15:272
1853899118,2cyclictest28696-21ssh22:56:012
1853899118,2cyclictest27295-21ssh23:54:032
1853899118,2cyclictest25552-21grep20:30:152
1853899118,2cyclictest22993-21sh23:31:272
1853899118,2cyclictest22706-21grep00:10:142
1853899118,2cyclictest21148-21basename19:15:132
1853899118,2cyclictest16808-21cat20:10:252
1853899118,2cyclictest11480-21ssh22:45:462
1853899118,2cyclictest10242-21cat21:05:172
1853899117,3cyclictest22735-21apt-get20:25:132
1853899113,5cyclictest24699-21apt-get22:35:142
1853899110,1cyclictest0-21swapper/223:42:282
1853899110,10cyclictest0-21swapper/222:20:472
1853699118,2cyclictest8073-21tune2fs21:00:161
1853699118,2cyclictest806-21wc00:35:251
1853699118,2cyclictest5598-21ssh23:20:551
1853699118,2cyclictest5119-21grep19:45:271
1853699118,2cyclictest3922-21grep20:50:201
1853699118,2cyclictest31740-21ssh22:20:041
1853699118,2cyclictest2732-21ssh22:40:271
1853699118,2cyclictest27093-21ssh23:15:151
1853699118,2cyclictest26360-21grep19:25:151
1853699118,2cyclictest21710-21tail21:15:181
1853699118,2cyclictest16863-21ssh21:50:521
1853699118,2cyclictest1530-21grep19:40:141
1853699118,2cyclictest10105-21grep21:05:161
1853699116,4cyclictest27205-21taskset20:32:181
1853699112,3cyclictest225-21systemd-journal20:56:441
1853699112,3cyclictest0-21swapper/122:15:131
1853699110,1cyclictest0-21swapper/122:32:521
1853399119,1cyclictest1640-21sort19:40:160
1853399118,2cyclictest8770-21sed22:45:000
1853399118,2cyclictest3495-21sh21:43:240
1853399118,2cyclictest31062-21sed21:20:270
1853399118,2cyclictest26079-21ssh22:55:140
1853399118,2cyclictest23981-21expr22:15:150
1853399118,2cyclictest2348-21ssh23:19:580
1853399118,2cyclictest22157-21ntpq21:15:220
1853399118,2cyclictest18330-21ssh23:48:340
1853399118,2cyclictest16661-21ssh21:12:100
1853399118,2cyclictest15767-21expr20:10:150
1853399118,2cyclictest14984-21wc00:05:190
1853399115,2cyclictest225-21systemd-journal19:19:590
1853399114,2cyclictest0-21swapper/023:00:130
18533991111,0cyclictest0-21swapper/022:01:350
1854399108,1cyclictest386-21ssh22:40:043
1854399108,1cyclictest3164-21df20:50:143
1854399108,1cyclictest25701-21cat20:30:153
1854399108,1cyclictest18388-21expr20:15:153
1854399108,1cyclictest15492-21cut20:10:133
1854399107,2cyclictest22960-21tr20:25:143
1854399107,2cyclictest19514-21head19:10:163
1854399107,2cyclictest15495-21ssh22:10:143
1854399107,2cyclictest1345-21cat19:40:143
1854399106,3cyclictest11316-21cat21:05:283
1854399105,4cyclictest28245-21latency_hist19:29:593
1853899108,2cyclictest4407-21uname19:45:182
1853899108,1cyclictest8631-21cat19:55:162
1853899108,1cyclictest28912-21apt-config23:55:002
1853899108,1cyclictest28912-21apt-config23:55:002
1853899107,2cyclictest8113-21cut21:00:162
1853899107,2cyclictest5276-21ls20:55:142
1853899103,3cyclictest17405-21apache_accesses22:50:122
1853899100,9cyclictest13360-21expr20:05:152
1853699108,1cyclictest31076-21sort20:40:191
1853699107,2cyclictest7009-21open_files19:50:191
1853699107,2cyclictest12240-21seq20:00:271
1853699101,4cyclictest0-21swapper/123:35:151
1853699100,9cyclictest18891-21cut20:15:191
1853699100,4cyclictest225-21systemd-journal19:37:271
1853399108,1cyclictest23275-21sed20:25:150
1853399108,1cyclictest20575-21taskset19:11:350
1853399107,2cyclictest6126-21tr19:50:130
1853399107,2cyclictest30479-21chrt19:31:010
1853399107,2cyclictest28887-21sed20:35:220
1853399107,2cyclictest27764-21tail19:25:280
1853399107,2cyclictest26119-21wc22:35:220
1853399107,2cyclictest1198-21taskset20:45:230
1853399102,4cyclictest225-21systemd-journal20:04:270
185439997,1cyclictest8534-21cat19:55:163
185389997,1cyclictest23858-21expr19:20:142
185369997,1cyclictest9915-21grep19:55:261
185369997,1cyclictest150792sleep120:09:531
185339997,1cyclictest7435-21acpi21:00:110
185339991,4cyclictest225-21systemd-journal20:45:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional